Dextrin

Copy Markdown View Source

CI Hex.pm Documentation

Dextrin is an Elixir implementation of DXN (Data eXchange Notation): a human-writable text format (.dxn), a compact binary format (.dxnb) built on CBOR, and a schema format (.dxns) that's just more DXN data — all three sharing one in-memory value representation and one extension mechanism.

{:ok, value} = Dextrin.decode(~s(%{x: 1, y: 2}))
value
#=> %{x: 1, y: 2}

Dextrin.encode(value)
#=> {:ok, "%{x:1,y:2}"}

{:ok, bytes} = Dextrin.encode_binary(value)
Dextrin.decode_binary(bytes)
#=> {:ok, %{x: 1, y: 2}}

A plain map's shorthand keys (x:) are themselves DXN keywords — %{x: 1} and %{:x => 1} are the exact same value, and keyword's own Elixir type, per DXN.md §1.3, is a real atom, which is what both decode/2 and decode_binary/2 produce by default (trusted: true). Encoding accepts a bare atom the same way, anywhere a keyword is expected — %{x: 1} or Dextrin.encode(:ok) both work with no conversion needed first.

That default assumes a source you control — your own config, your own application's data — not arbitrary untrusted/network input, where an unbounded String.to_atom/1 could exhaust the atom table. Pass trusted: false for that case; keyword then decodes as Dextrin.Keyword.t() instead:

Dextrin.decode(~s(%{x: 1, y: 2}), trusted: false)
#=> {:ok, %{%Dextrin.Keyword{name: "x"} => 1, %Dextrin.Keyword{name: "y"} => 2}}

A schema-backed struct's fields are the one place names do come back as plain strings regardless of trusted: — see the tutorial — since a schema always knows its field names up front.

.dxn's text grammar is compiled by Ichor — write the grammar once (priv/grammar/dxn.aether), get a lexer, parser, and (via Dextrin.Text.Actions) an evaluator with no hand-written parsing code. That compilation happens ahead of time (mix ichor.gen, checked in as generated source), not at dextrin's own build time, so only the small ichor_runtime support library the generated code actually calls ships as a real dependency — ichor proper (the Aether front-end, analysis, codegen) is dev-tooling only. .dxnb has no grammar to speak of — it's a direct, hand-rolled CBOR codec — so it's plain recursive Elixir working over the same shared value type.

Why

Most serialization formats pick one point on a spectrum: JSON is human-writable but loses precision (no distinct int/float boundary, no dates, no bytes) and has no extension story; Protobuf/Avro are compact and typed but need a separate schema-compiler step and aren't meant for a human to read or hand-edit; EDN is expressive and Elixir-friendly in spirit but has no first-party Elixir implementation and no binary counterpart. DXN's premise is that a .dxn text document and a .dxnb binary document should be the same value space — 30 scalar/collection/temporal/extended types (reference §1.3), precise enough for money (Decimal), exact ratios (Rational), and arbitrary precision integers, with struct and custom-tag as first-class, schema-describable extension points — encoded however density or readability happens to matter for a given use.

Components

  • Dextrin — the four-function public API: decode/2, encode/2 (.dxn text) and decode_binary/2, encode_binary/2 (.dxnb binary). One error type, Dextrin.Error, for both.
  • Value types (Dextrin.Symbol, Dextrin.Tuple, Dextrin.OrderedMap, Dextrin.SortedSet, Dextrin.Struct, Dextrin.Array, Dextrin.Duration, Dextrin.Rational, Dextrin.Uuid, Dextrin.Uri, Dextrin.Bytes, Dextrin.Char, Dextrin.CustomTag) — small wrapper structs for the DXN types Elixir has nothing native for without losing information. keyword is the one exception with two faces: a real atom by default (trusted: true), Dextrin.Keyword when decoded untrusted. Every other scalar/collection (integers, floats, strings, lists, plain maps, sets, dates, regexes, ...) decodes to the obvious native Elixir value.
  • Dextrin.Text.Grammar/Actions/Printer/Formatter — the .dxn pipeline: an Ichor-compiled grammar (Grammar is a thin wrapper around the pregenerated Grammar.Native), an Ichor.Actions implementation that turns a parse into real values, a single-line printer (the reverse direction), and a multi-line pretty-formatter on top of it.
  • Dextrin.Binary.Encoder/Decoder/Tags — the .dxnb pipeline: a direct recursive CBOR codec (not built on a generic CBOR library — see Dextrin.Binary.Encoder's own moduledoc for why) plus the private tag block and bit-layout constants it needs.
  • Dextrin.Schema and Dextrin.Schema.* — compiles a .dxns document (itself just DXN data — no new grammar) into a Dextrin.Registry, enforced automatically, decode- and encode-side, wherever a registered struct name appears. Dextrin.Schema.Std ships a small standard library of common named types (PositiveInteger, NonEmptyString, ...); Dextrin.Schema.FileResolver resolves Namespace/Name references across separate .dxns files; Dextrin.Schema.Provider lets a struct's own library ship its DXN schema without that library ever depending on dextrin itself.
  • Dextrin.Registry — the one extension point both struct and custom-tag share: register a tag decoder/encoder, a struct materializer, or a lazy schema resolver; also carries the trusted:/put_trusted/2 flag that decides how keyword decodes. Plain immutable data, threaded explicitly — never a process or ETS table.
  • mix dextrin.* — validate, encode, decode, format, gen.schema (scaffold a .dxns file from an existing Elixir struct), and gen.unicode (regenerate the grammar's Unicode identifier ranges from the latest UCD data).

Installation

Add dextrin to your list of dependencies in mix.exs:

def deps do
  [
    {:dextrin, "~> 0.1.0"}
  ]
end

Where to go next

  • Tutorial — a step-by-step walkthrough of this library's features, building up to a small, working example that decodes, validates against a schema, and re-encodes real data.
  • Examples — worked examples: config files, API payloads, event logs, and schema-validated records.
  • Cheatsheet — quick reference for common Dextrin tasks.
  • DXN tutorial and DXN reference — everything about the DXN format itself, independent of this Elixir implementation, plus DXN examples and a DXN cheatsheet.

Development

mix deps.get
mix precommit

mix precommit runs the full verification pass this project expects before a commit: mix format, mix compile --warnings-as-errors, mix credo --strict, mix sobelow, mix test, and mix dialyzer, in that order (fast/cheap checks first, dialyzer — the slowest, especially its first PLT build — last).

priv/grammar/dxn.aether's generated Unicode identifier ranges are regenerated with mix dextrin.gen.unicode — a deliberate, reviewed action on a Unicode version bump, never run automatically at build time (see that task's own docs). Either way, changing the grammar itself requires a mix ichor.gen step afterward, since lib/dextrin/text/grammar/native.ex is generated ahead of time, not produced at dextrin's own compile time (that's also what keeps ichor itself, and everything it depends on for parsing/codegen, only: :dev, runtime: false — only the small ichor_runtime package ships in a release) — see CONTRIBUTION.md.

See CONTRIBUTION.md for how to propose changes, and CHANGELOG.md for release history.

License

MIT — see LICENSE.