Dextrin

Dextrin is an Elixir implementation of DXN (Data eXchange Notation): a human-writable text format (.dxn), a compact binary format (.dxnb) built on CBOR, and a schema format (.dxns) that's just more DXN data — all three sharing one in-memory value representation and one extension mechanism.

This is the reference implementation. node-dextrin (Node.js, on npm) and php-dextrin (PHP, on Packagist) are sibling ports of the same format — see Other language implementations below.

{:ok, value} = Dextrin.decode(~s(%{x: 1, y: 2}))
value
#=> %{x: 1, y: 2}

Dextrin.encode(value)
#=> {:ok, "%{x:1,y:2}"}

{:ok, bytes} = Dextrin.encode_binary(value)
Dextrin.decode_binary(bytes)
#=> {:ok, %{x: 1, y: 2}}

A plain map's shorthand keys (x:) are themselves DXN keywords — %{x: 1} and %{:x => 1} are the exact same value, and keyword's own Elixir type, per DXN.md §1.3, is a real atom, which is what both decode/2 and decode_binary/2 produce by default (trusted: true). Encoding accepts a bare atom the same way, anywhere a keyword is expected — %{x: 1} or Dextrin.encode(:ok) both work with no conversion needed first.

That default assumes a source you control — your own config, your own application's data — not arbitrary untrusted/network input, where an unbounded String.to_atom/1 could exhaust the atom table. Pass trusted: false for that case; keyword then decodes as Dextrin.Keyword.t() instead:

Dextrin.decode(~s(%{x: 1, y: 2}), trusted: false)
#=> {:ok, %{%Dextrin.Keyword{name: "x"} => 1, %Dextrin.Keyword{name: "y"} => 2}}

A schema-backed struct's fields are the one place names do come back as plain strings regardless of trusted: — see the tutorial — since a schema always knows its field names up front.

.dxn's text grammar is compiled by Ichor — write the grammar once (priv/grammar/dxn.aether), get a lexer, parser, and (via Dextrin.Text.Actions) an evaluator with no hand-written parsing code. That compilation happens ahead of time (mix ichor.gen, checked in as generated source), not at dextrin's own build time, so only the small ichor_runtime support library the generated code actually calls ships as a real dependency — ichor proper (the Aether front-end, analysis, codegen) is dev-tooling only. .dxnb has no grammar to speak of — it's a direct, hand-rolled CBOR codec — so it's plain recursive Elixir working over the same shared value type.

Why use Dextrin?

Most serialization formats pick one point on a spectrum. JSON is human-writable but has no dates, no bytes, no exact decimals and no boundary between an integer and a float. Protobuf and Avro are compact and typed but need a schema compiled ahead of time, and aren't meant for a person to read or edit. CBOR and MessagePack are compact and self-describing but binary only. YAML and TOML are made for people, and have no binary form for machines.

DXN's premise is that a .dxn text document and a .dxnb binary document are the same value space: 30 scalar, collection, temporal and extended types (reference §1.3). It is precise enough for money (decimals), exact ratios (rationals) and integers of any size, has struct and custom-tag as first-class, schema-describable extension points, and its schemas (.dxns) are DXN too. You choose text or binary by what matters where the data is going -- a person reading it, or bytes on the wire -- and the value is the same either way.

What each format can say

¹ CBOR has a diagnostic notation for reading, not for authoring. ² Protobuf's text format is for debugging; its schema is a separate .proto file compiled ahead of time. ³ JSON Schema and Ion Schema are written in JSON and Ion, as separate specifications. ⁴ The grammar allows any size; JavaScript's and PHP's parsers lose the digits past 53 and 64 bits. ⁵ YAML 1.1 tags (!!timestamp, !!binary, !!set), dropped from 1.2's core schema. ⁶ Dates, times and date-times; no durations. ⁷ An instant only: no local dates, times or durations. ⁸ The well-known Timestamp and Duration messages. ⁹ A UUID as binary subtype 4. ¹⁰ Sets by registered tag 258; no tuples. ¹¹ Integer and string keys only. ¹² Field numbers keep old readers working; nothing checks a change. ¹³ Instants (tags 0 and 1) and dates (RFC 8943); no times of day or durations. ¹⁴ Timestamps of any precision, dates included; no times of day or durations.

How big it is

The same four payloads in each format, in bytes, raw and gzipped. Reproduce it with mix run bench/format_sizes.exs; the script says how each format writes the values it has no type for (a decimal or a date as a string in JSON, for instance).

• DXN text is about JSON's size -- keys need no quotes -- while carrying the types JSON drops.
• .dxnb sits between JSON and the compact binaries. Value sharing (encode_binary(value, share: true)) writes a repeated key or string once; on the API response that brings it within 7% of Protocol Buffers, with no schema needed to read it.
• Where it is larger, it is carrying more. A .dxnb timestamp keeps microseconds and its type, where CBOR's and MessagePack's carry whole seconds; a keyword key is tagged as a keyword rather than written as a plain string. Protocol Buffers is smallest because its schema, not the data, holds the field names.
• Compressed, the gap narrows. Gzipped, DXN text stays within about 12% of .dxnb, and JSON within about a quarter of CBOR; past that, the choice is about what each format can say.

What comes back, in each language

What a value is after a round trip through each language's own JSON parser, and through Dextrin:

A bold entry is a value that did not come back as what was written. With Dextrin, the same document means the same thing in all four, and every implementation is held to that by the Dextrin Conformance Suite.

Components

• Dextrin — the four-function public API: decode/2, encode/2 (.dxn text) and decode_binary/2, encode_binary/2 (.dxnb binary). One error type, Dextrin.Error, for both.
• Value types (Dextrin.Symbol, Dextrin.Tuple, Dextrin.OrderedMap, Dextrin.SortedSet, Dextrin.Struct, Dextrin.Array, Dextrin.Duration, Dextrin.Rational, Dextrin.Uuid, Dextrin.Uri, Dextrin.Bytes, Dextrin.Char, Dextrin.CustomTag) — small wrapper structs for the DXN types Elixir has nothing native for without losing information. keyword is the one exception with two faces: a real atom by default (trusted: true), Dextrin.Keyword when decoded untrusted. Every other scalar/collection (integers, floats, strings, lists, plain maps, sets, dates, regexes, ...) decodes to the obvious native Elixir value.
• Dextrin.Text.Grammar/Actions/Printer/Formatter — the .dxn pipeline: an Ichor-compiled grammar (Grammar is a thin wrapper around the pregenerated Grammar.Native), an Ichor.Actions implementation that turns a parse into real values, a single-line printer (the reverse direction), and a multi-line pretty-formatter on top of it.
• Dextrin.Binary.Encoder/Decoder/Tags — the .dxnb pipeline: a direct recursive CBOR codec (not built on a generic CBOR library — see Dextrin.Binary.Encoder's own moduledoc for why) plus the private tag block and bit-layout constants it needs.
• Dextrin.Schema and Dextrin.Schema.* — compiles a .dxns document (itself just DXN data — no new grammar) into a Dextrin.Registry, enforced automatically, decode- and encode-side, wherever a registered struct name appears. Dextrin.Schema.Std ships a small standard library of common named types (PositiveInteger, NonEmptyString, ...); Dextrin.Schema.FileResolver resolves Namespace/Name references across separate .dxns files; Dextrin.Schema.Provider lets a struct's own library ship its DXN schema without that library ever depending on dextrin itself.
• Dextrin.Registry — the one extension point both struct and custom-tag share: register a tag decoder/encoder, a struct materializer, or a lazy schema resolver; also carries the trusted:/put_trusted/2 flag that decides how keyword decodes. Plain immutable data, threaded explicitly — never a process or ETS table.
• mix dextrin.* — validate, encode, decode, format, compat (the schema version discipline check, for CI), gen.schema (scaffold a .dxns file from an existing Elixir struct), and gen.unicode (regenerate the grammar's Unicode identifier ranges from the latest UCD data).

Installation

Add dextrin to your list of dependencies in mix.exs:

def deps do
  [
    {:dextrin, "~> 0.2.0"}
  ]
end

Other language implementations

DXN's format spec (guides/dxn/DXN.md) is implementation-independent — this library, node-dextrin, and php-dextrin all satisfy the same document, and stay close to this project's own in-memory shape-per-type choices wherever their host language allows, so recognizing one implementation gets you most of the way to recognizing the others.

Where to go next

• Tutorial — a step-by-step walkthrough of this library's features, building up to a small, working example that decodes, validates against a schema, and re-encodes real data.
• Examples — worked examples: config files, API payloads, event logs, and schema-validated records.
• Cheatsheet — quick reference for common Dextrin tasks.
• DXN tutorial and DXN reference — everything about the DXN format itself, independent of this Elixir implementation, plus DXN examples and a DXN cheatsheet.

Development

mix deps.get
mix precommit

mix precommit runs the full verification pass this project expects before a commit: mix format, mix compile --warnings-as-errors, mix credo --strict, mix sobelow, mix test, and mix dialyzer, in that order (fast/cheap checks first, dialyzer — the slowest, especially its first PLT build — last).

priv/grammar/dxn.aether's generated Unicode identifier ranges are regenerated with mix dextrin.gen.unicode — a deliberate, reviewed action on a Unicode version bump, never run automatically at build time (see that task's own docs). Either way, changing the grammar itself requires a mix ichor.gen step afterward, since lib/dextrin/text/grammar/native.ex is generated ahead of time, not produced at dextrin's own compile time (that's also what keeps ichor itself, and everything it depends on for parsing/codegen, only: :dev, runtime: false — only the small ichor_runtime package ships in a release) — see CONTRIBUTING.md.

See CONTRIBUTING.md for how to propose changes, and CHANGELOG.md for release history.

License

Apache-2.0 — see LICENSE.