Glazer

Very fast Erlang NIF encoder/decoder for JSON, YAML, and CSV, built around hand-rolled recursive-descent decoders and direct term-to-text encoders that produce/consume native Erlang terms in a single pass. The JSON implementation was inspired by the glaze C++ library; glazer has since matured into a standalone implementation with no external C++ dependencies, and extended the same approach to YAML and CSV, with performance and features unmatched by other existing libraries for these formats.

Performance

• JSON: faster encoding than every other library benchmarked, and roughly on par with torque (Rust sonic-rs NIF) on decoding — both well ahead of simdjsone, jiffy, and the pure-Elixir libraries jason, thoas, euneus, and OTP's built-in json.
• YAML: an order of magnitude faster than yaml_rustler and fast_yaml, and ~10-100x faster than the pure-Erlang yamerl/ymlr.
• CSV: 2-20x faster than nimble_csv, and tens to hundreds of times faster than csv and erl_csv (which times out on large inputs).

Each chart compares glazer against other libraries for JSON/YAML/CSV decode and encode on a representative small/medium/large file. Charts are generated from the tables below via scripts/gen_bench_charts.py. Benchmark tables:

• Benchmarking JSON
• Benchmarking YAML
• Benchmarking CSV

Features

JSON

• Decoding straight to Erlang terms: maps, lists, binaries, integers (including bignums), floats, booleans, and null
• Encoding Erlang terms straight to JSON, including big integers
• Incremental/streaming decoding of partial input (e.g. NDJSON over a socket) via json_stream_decoder/0,1, json_stream_feed/2, json_stream_eof/1
• Configurable representation of JSON null and JSON object keys
• json_minify/1 and json_prettify/1 helpers
• Standalone big-integer encode/decode helpers (encode_integer/1, decode_integer/1, try_decode_integer/1)
• json_query/2,3: run a jq filter over a JSON document, returning decoded Erlang terms (requires glazer to be built with libjq available — see jq filter support)

YAML

• Decoding YAML mappings/sequences/scalars to Erlang maps/lists/scalars, including big integers
• Encoding Erlang terms to YAML in block style
• Configurable representation of YAML null and mapping keys, with optional YAML 1.1 boolean compatibility (yes/no/on/off)

CSV

• RFC 4180 CSV encoding/decoding via csv_decode/1,2 and csv_encode/1,2, with optional header-row support
• Incremental/streaming CSV decoding via csv_stream_decoder/0,1, csv_stream_feed/2, csv_stream_eof/1

Scope

glazer targets formats that map naturally onto a tree of Erlang maps/lists/scalars — JSON and YAML both fit this model directly, so a single decode/encode pair can convert losslessly between the format and native terms. XML is intentionally not planned: its data model (tagged elements, attributes, mixed text/element content, namespaces, processing instructions, entities) has no single natural Erlang term representation, and any choice (xmerl-style tuples, JSON-like maps with @attr/#text keys, etc.) is a lossy or awkward fit compared to formats that are already trees of scalars and collections. Erlang's standard library already ships xmerl for XML; there's little value in duplicating it here with a different, opinionated term shape.

Installation

Erlang (rebar.config):

{deps, [
  {glazer, "~> 0.3"}
]}.

Elixir (mix.exs):

def deps do
  [
    {:glazer, "~> 0.3"}
  ]
end

Building

Building the NIF requires a C++23 compiler (GCC 12+ or Clang 16+) and make. There are no external C++ library dependencies — all C++ code is self-contained in c_src/. A plain

make

builds priv/glazer.so and compiles the Erlang sources. For the fastest performance, run a Profile-Guided Optimisation (PGO) build instead:

make optimize

This performs three steps automatically: compiles an instrumented binary, runs the test suite to collect real branch-frequency data, then recompiles with those profiles applied. The resulting .so typically outperforms a plain -O3 build by 5–15% on realistic JSON workloads.

glazer is an Erlang application with a Rebar-based C++ NIF build; mix invokes the same top-level Makefile/rebar3 compile path described above, so the same C++23 compiler requirement applies. Once compiled, call it via the :glazer module from Elixir:

Erlang:

1> glazer:json_decode(~"{\"a\":1,\"b\":[true,null,3.5]}")
#{<<"a">> => 1,<<"b">> => [true,null,3.5]}

Elixir:

iex> :glazer.json_encode(%{"a" => 1, "b" => [true, :null, 3.5]})
"{\"a\":1,\"b\":[true,null,3.5]}"

Use the use_nil/{null_term, nil} option (see JSON null below) to get idiomatic Elixir nil instead of the atom :null.

JSON

Usage

1> glazer:json_decode(<<"{\"a\":1,\"b\":[true,null,3.5]}">>).
#{<<"a">> => 1, <<"b">> => [true, null, 3.5]}

2> glazer:json_encode(#{<<"a">> => 1, <<"b">> => [true, null, 3.5]}).
<<"{\"a\":1,\"b\":[true,null,3.5]}">>

3> glazer:json_encode(#{a => 1}, [pretty]).
<<"{\n  \"a\": 1\n}">>

4> glazer:json_minify(<<" { \"a\" : 1 } ">>).
{ok, <<"{\"a\":1}">>}

5> glazer:json_prettify(<<"{\"a\":1}">>).
{ok, <<"{\n  \"a\": 1\n}">>}

Streaming

For input that arrives in chunks — e.g. reading a large document incrementally, or consuming newline-delimited JSON (NDJSON) from a socket or file — json_stream_decoder/0,1 provides a small stateful wrapper that buffers partial input and decodes each JSON value as soon as it's complete, without re-parsing bytes you've already seen:

1> D0 = glazer:json_stream_decoder(),
2> {Vals1, D1} = glazer:json_stream_feed(D0, <<"{\"a\":1} {\"b\":">>),
3> Vals1.
[#{<<"a">> => 1}]

4> {Vals2, D2} = glazer:json_stream_feed(D1, <<"2}">>),
5> Vals2.
[#{<<"b">> => 2}]

6> glazer:json_stream_eof(D2).
{ok, []}

json_stream_feed/2 returns the list of values completed by the chunk just fed (possibly empty, possibly more than one if the chunk completes several values) along with the updated decoder state to pass to the next call. Once the input is exhausted, call json_stream_eof/1 to flush any trailing bare scalar (numbers, strings, etc. have no closing delimiter of their own) and surface an error if the buffer holds an incomplete value:

1> D0 = glazer:json_stream_decoder(),
2> {[], D1} = glazer:json_stream_feed(D0, <<"   42">>),
3> glazer:json_stream_eof(D1).
{ok, [42]}

json_stream_decoder/1 accepts the same options as json_decode/2 (e.g. {keys, atom}, use_nil) and applies them to every decoded value.

A typical read loop calls json_stream_feed/2 for each chunk while more data may still arrive, and json_stream_eof/1 once the socket closes to flush any trailing value:

loop(Socket, D0) ->
  case gen_tcp:recv(Socket, 0) of
    {ok, Chunk} ->
      {Vals, D1} = glazer:json_stream_feed(D0, Chunk),
      handle_values(Vals),
      loop(Socket, D1);
    {error, closed} ->
      case glazer:json_stream_eof(D0) of
        {ok, Trailing}  -> handle_values(Trailing);
        {error, Reason} -> handle_truncated_stream(Reason)
      end
  end.

Efficiency

json_stream_feed/2 only scans for value boundaries incrementally — the scanner carries a small resumable cursor (scan_state()) that remembers how far it has already looked (nesting depth, whether it's inside a string, escape state, …), so each call to json_scan/2 resumes from where the previous one left off rather than re-walking the whole buffer from byte zero. Once a complete value's end offset is known, that slice is decoded exactly once via the same NIF-backed decoder used by json_decode/2 — there's no intermediate tokenization or tree representation, and no byte is ever scanned or decoded twice. The only buffering cost is concatenating newly-arrived chunks onto the not-yet-complete tail of the input.

This makes json_stream_feed/2 well suited to byte-at-a-time or small-chunk feeding (e.g. consuming a gen_tcp/gen_statem socket buffer as it fills) without the quadratic-rescan cost a naive "concatenate and retry full decode" loop would incur on large or slow-arriving documents.

Under the hood, json_stream_feed/2 is built on json_scan/1,2 — a low-level primitive that scans a buffer for the byte offset where the next JSON value ends (or reports that more input is needed) without doing a full decode. It's exposed directly for callers that want to implement their own framing/buffering strategy:

1> glazer:json_scan(<<"{\"a\":1} {\"b\":2}">>).
{complete, 7}

2> glazer:json_scan(<<"{\"a\":">>).
{incomplete, ScanState}

3> glazer:json_scan(<<"{\"a\":1}">>, ScanState).
{complete, 7}

json_stream_decoder/0,1, json_stream_feed/2, json_stream_eof/1 and json_scan/1,2 are JSON-only — see YAML streaming and CSV streaming below for the other formats.

JSON null

By default, JSON null decodes to (and null encodes from) the atom null. This can be overridden:

• Application-wide, via the null environment key — set this once in the application's config and every call uses it as the default:

  Erlang (rebar.config):

  {glazer, [{null, nil}]}
  

  Elixir (config.exs):

  config :glazer, null: nil
  

• Per call, with the use_nil shorthand or the {null_term, Atom} option (see Decode options below). Per-call options always take precedence over the application-wide default.

Decode options (json_decode/2)

1> glazer:json_decode(<<"{\"a\":1}">>, [object_as_tuple]).
{[{<<"a">>, 1}]}

2> glazer:json_decode(<<"{\"a\":1}">>, [{keys, atom}]).
#{a => 1}

3> glazer:json_decode(<<"null">>, [use_nil]).
nil

4> glazer:json_decode(<<"null">>, [{null_term, undefined}]).
undefined

5> glazer:json_decode(<<"{\"a\":1,\"a\":2}">>).
#{<<"a">> => 2}

6> glazer:json_decode(<<"{\"a\":1,\"a\":2}">>, [object_as_tuple]).
{[{<<"a">>, 1}, {<<"a">>, 2}]}

7> glazer:json_decode(<<"{\"a\":1,\"a\":2}">>, [object_as_tuple, dedupe_keys]).
{[{<<"a">>, 2}]}

Encode options (json_encode/2)

1> glazer:json_encode(#{a => 1}, [pretty]).
<<"{\n  \"a\": 1\n}">>

2> glazer:json_encode(<<"héllo"/utf8>>, [uescape]).
<<"\"h\\u00e9llo\"">>

3> glazer:json_encode(nil, [use_nil]).
<<"null">>

jq filter support

If libjq and its headers (jq.h/jv.h) are available when glazer is built, json_query/2,3 runs a jq filter program against a JSON document and returns one Erlang term per value produced by the filter (decoded using the same options as json_decode/2):

1> glazer:json_query(<<"{\"a\":[1,2,3]}">>, <<".a[]">>).
{ok, [1, 2, 3]}

2> glazer:json_query(<<"{\"a\":1}">>, <<".b">>).
{ok, [null]}

3> glazer:json_query(<<"{\"a\":{\"b\":2}}">>, <<".">>, [{keys, atom}]).
{ok, [#{a => #{b => 2}}]}

4> glazer:json_query(<<"not json">>, <<".">>).
{error, invalid_input}

5> glazer:json_query(<<"{\"a\":1}">>, <<"bad syntax (((">>).
{error, jq_decode_error}

If libjq was not available at build time, json_query/2,3 returns {error, jq_not_available}. Build detection is automatic — make probes for jq.h/libjq and only enables this feature if found, so glazer still builds and works without libjq installed.

API

Benchmarking JSON

A comparison benchmark against other JSON libraries (simdjsone, jiffy, jason, thoas, euneus, OTP's built-in json, and torque) is available via:

$ PARALLEL=2 make bench
==> Running benchmarks with parallelism: 2

(numbers in µs)
JSON        twitter (616.7K)   twitter2 (758.0K)     openrtb (1.2K)       esad (1.3K)         small (0.1K)
            decode   encode     decode   encode     decode   encode     decode   encode     decode   encode
-------------------------------------------------------------------------------------------------------------
glazer      3876.2   1205.8     4716.9   1971.3        9.1      3.7        5.8      2.9        0.9      0.8
torque      5140.3   1333.7     4468.8   4411.4        9.3      5.5        4.9      3.5        1.8      1.4
simdjsone   4652.9   3564.3     7426.9   6236.6       10.2     13.5        7.7      8.4        1.2      2.1
jiffy       5921.4   2547.3     7974.6   4654.6       11.1     11.4        7.9      6.2        1.8      1.9
jason      10040.8   8237.9    18594.6  17591.8       23.4     21.9       16.3     20.6        2.8      2.2
thoas      10245.4   9117.9    19040.7  18598.2       23.7     23.5       19.6     21.2        2.6      2.3
euneus     10446.2   6845.7    14069.1  12934.7       20.8     15.5       11.9      9.5        3.1      2.1
json       10061.2   6563.0    13467.7  12629.6       19.6     16.0       11.1      8.3        2.5      1.8

(requires the bench/dev Mix dependencies — see mix.exs).

Performance

glazer has a faster JSON encoder than all competitors. glazer is roughly on par with torque (a Rust sonic-rs NIF) across the benchmarked workloads on decoding — neither library is consistently faster, and the gap on any given file/operation is typically modest (within ~30%), varying in direction from file to file. Both sit well ahead of the other contenders (simdjsone, jiffy, and the pure-Elixir libraries jason, thoas, euneus, and OTP's built-in json).

Where glazer has an edge over torque:

• No tuple-of-binaries intermediate representation.glazer decodes straight to native Erlang terms (maps, lists, binaries, numbers) and encodes straight from them, in a single pass, with no generic JSON-tree staging step — minimizing allocation and copying on both the decode and encode paths.
• Big integer support. JSON numbers that overflow 64 bits decode to Erlang bignums (and encode back to their exact decimal form) — see Big integers. torque does not support this.
• Configurable null and object-key representation.null_term/use_nil and {keys, atom | existing_atom | binary} let you tailor the decoded shape to your application without a post-processing pass.
• uescape/force_utf8 encode options for \uXXXX-escaping non-ASCII output and sanitizing invalid UTF-8 — useful when targeting strict JSON consumers or transports that aren't UTF-8 clean.
• Standalone json_minify/1/json_prettify/1 and big-integer helpers (encode_integer/1/decode_integer/1/try_decode_integer/1) that don't require a full decode/encode round-trip.
• No external C++ dependencies. The NIF is fully self-contained — no CMake, no FetchContent, no vendored third-party library to pull at build time — vs. torque's reliance on a Rust toolchain and sonic-rs, which adds a second language/toolchain to the build.

Performance optimizations

A few implementation techniques in c_src/glazer_nif.cpp account for most of the gap over the slower contenders:

• Single-pass, zero-copy decode/encode. As noted above, there's no intermediate generic JSON tree — the decoder builds Erlang terms directly from the input bytes (string keys/values are views into the original binary whenever no escaping is needed) and the encoder writes JSON bytes directly from Erlang terms. This removes a whole staging allocate-and-copy pass that tree-based decoders pay for.

• Inline, growable output buffer (OutBuf). Encoding writes into a 4 KB stack-allocated buffer first; only documents that exceed that spill to the heap, growing geometrically via malloc/realloc (the latter resizes in place when possible, avoiding a copy on every growth — a plain new[]/delete[] doubling strategy can't do this).

• Key cache for repeated object keys (KeyCache). Real-world JSON documents reuse the same small set of key strings heavily (e.g. a Twitter feed has ~13K key occurrences across only ~94 distinct keys). KeyCache is an open-addressed hash table (power-of-two size, linear probing, FNV-1a hash with a precomputed-hash fast-reject before the memcmp) that lets a repeated key reuse the same already-built ERL_NIF_TERM binary instead of paying enif_make_new_binary + memcpy again. It's only engaged for inputs above a size threshold (KEY_CACHE_MIN_SIZE), since small payloads (RPC-sized messages) rarely repeat keys enough to amortize the lookup cost.

• Epoch-counter lazy clearing. Both KeyCache and the scratch buffers it touches need to start "empty" on every decode call, but zero-initializing a multi-KB table for every single call — including tiny documents that never populate it — would cost more than the cache saves. Instead each cache entry carries a generation/epoch tag; a slot is considered live only if its epoch matches the cache's current m_epoch (itself seeded from a process-wide monotonically-increasing counter, so leftover garbage from a prior stack frame can never coincidentally look live). This makes cache construction effectively free, regardless of table size.

• SWAR whitespace skipping.skip_ws checks the next byte before paying for any wider load, then — for runs of whitespace — scans 8 bytes at a time using branch-free bit-twiddling ("SIMD within a register") to find the first non-whitespace byte, rather than testing one byte at a time. Minified JSON (the overwhelmingly common case) has little or no structural whitespace, so the single-byte fast path dominates in practice.

• Table-driven string escaping with bulk copies. JSON string escaping scans for runs of bytes that need no escaping (a precomputed 256-entry lookup table answers "does this byte need escaping?" in O(1)) and copies each run in one memcpy, falling into a per-byte switch only for the rare characters that actually need an escape sequence.

• Fast integer formatting. Integers are written to JSON using a lookup-table-based digit-pair algorithm (avoiding division for small values) with a vendored lltoa fallback for larger numbers — faster than routing every integer through snprintf.

YAML

Usage

yaml_decode/1,2 decodes a YAML document to an Erlang term — mappings become maps, sequences become lists, and scalars become the matching Erlang type (binaries, numbers, booleans, or null):

1> glazer:yaml_decode(<<"a: 1\nb:\n  - true\n  - null\n  - 3.5\n">>).
#{<<"a">> => 1, <<"b">> => [true, null, 3.5]}

2> glazer:yaml_encode(#{<<"a">> => 1, <<"b">> => [true, null, 3.5]}).
<<"a: 1\nb:\n  - true\n  - null\n  - 3.5\n">>

yaml_encode/1,2 encodes an Erlang term to YAML in block style (2-space indentation, sequences at the same indentation as the mapping key that owns them).

Streaming

There is no incremental YAML decoder. YAML's block styles have no closing delimiter — a mapping or sequence simply ends at a dedent or end-of-input — so there is no way to scan a partial buffer for "is this value complete yet?" the way json_scan/1,2 does for JSON's bracket-balanced syntax. Decode full YAML documents with yaml_decode/1,2 once they are fully buffered.

Decode options (yaml_decode/2)

1> glazer:yaml_decode(<<"a: ~\n">>, [use_nil]).
#{<<"a">> => nil}

2> glazer:yaml_decode(<<"a: 1\n">>, [{keys, atom}]).
#{a => 1}

3> glazer:yaml_decode(<<"a: yes\n">>, [yaml_1_1_bools]).
#{<<"a">> => true}

Encode options (yaml_encode/2)

1> glazer:yaml_encode(#{<<"a">> => nil}, [use_nil]).
<<"a: null\n">>

API

Benchmarking YAML

$ PARALLEL=2 make bench-yaml
==> Running benchmarks with parallelism: 2

(numbers in µs)
YAML             openrtb (1.3K)       esad (1.3K)         small (0.1K)
                decode   encode     decode   encode     decode   encode
-------------------------------------------------------------------------
glazer            33.2      7.7       27.2      5.3        7.9      1.2
yaml_rustler     126.5      n/a       78.0      n/a       11.8      n/a
fast_yaml        180.0     45.1       97.2     40.6       18.8      6.1
yamerl          1366.2      n/a      991.5      n/a      517.3      n/a
ymlr               n/a     47.4        n/a     35.3        n/a      4.8

CSV

Usage

csv_decode/1,2 decodes an RFC 4180 CSV document to a list of rows, each row a list of binary fields:

1> glazer:csv_decode(<<"name,age\nAlice,30\nBob,25\n">>).
[[<<"name">>, <<"age">>], [<<"Alice">>, <<"30">>], [<<"Bob">>, <<"25">>]]

2> glazer:csv_encode([[<<"name">>, <<"age">>], [<<"Alice">>, 30]]).
<<"name,age\r\nAlice,30\r\n">>

With the headers option, the first row is used as column names and each subsequent row decodes to a map; csv_encode/2 with headers does the reverse, deriving the header row from the first map's keys:

1> glazer:csv_decode(<<"name,age\nAlice,30\n">>, [headers]).
[#{<<"name">> => <<"Alice">>, <<"age">> => <<"30">>}]

2> glazer:csv_encode([#{<<"name">> => <<"Alice">>, <<"age">> => 30}], [headers]).
<<"name,age\r\nAlice,30\r\n">>

Fields containing the delimiter, a double quote, or a line break are quoted automatically on encode (with embedded quotes doubled), and unquoted on decode. The delimiter defaults to , and can be changed via {delimiter, Char}; the encoded line ending defaults to \r\n per RFC 4180 and can be changed to \n via {line_ending, lf}.

Streaming

For input that arrives in chunks, csv_stream_decoder/0,1 provides the same kind of stateful wrapper as JSON streaming: it buffers partial input and decodes each row as soon as its terminating line break is seen, via csv_decode/2 on that single row. A small scanner tracks whether the cursor is inside a quoted field across chunks, so a \n/\r\n inside a quoted field doesn't end the row:

1> D0 = glazer:csv_stream_decoder(),
2> {Rows1, D1} = glazer:csv_stream_feed(D0, <<"a,b\n1,2\n3,">>),
3> Rows1.
[[<<"a">>,<<"b">>],[<<"1">>,<<"2">>]]

4> {Rows2, D2} = glazer:csv_stream_feed(D1, <<"4\n">>),
5> Rows2.
[[<<"3">>,<<"4">>]]

6> glazer:csv_stream_eof(D2).
{ok, []}

csv_stream_feed/2 returns the rows completed by the chunk just fed (possibly empty, possibly more than one) along with the updated decoder state. Once the input is exhausted, call csv_stream_eof/1 to flush a trailing row that has no terminating line break, or surface an error if the buffered bytes don't form a valid row:

1> D0 = glazer:csv_stream_decoder(),
2> {Rows1, D1} = glazer:csv_stream_feed(D0, <<"a,b\n1,2">>),
3> Rows1.
[[<<"a">>,<<"b">>]]

4> glazer:csv_stream_eof(D1).
{ok, [[<<"1">>,<<"2">>]]}

csv_stream_decoder/1 accepts the same options as csv_decode/2. With the headers option, the first complete row is captured as the header and used to decode every subsequent row as a map; no row is emitted for the header itself. Blank lines are skipped, matching csv_decode/2.

Decode options (csv_decode/2)

Encode options (csv_encode/2)

API

Benchmarking CSV

$ PARALLEL=2 make bench-csv
==> Running benchmarks with parallelism: 2

(numbers in µs)
CSV               small (1.3K)          medium (130.9K)         large (3433.1K)
                decode     encode       decode     encode       decode     encode
-----------------------------------------------------------------------------------
glazer             8.3        4.9       1066.0      380.7      38298.9    12887.9
nimble_csv        33.9       23.4       4202.3     2535.2     173790.7    94933.2
csv               75.1      177.9       6250.8    15820.6      TIMEOUT    TIMEOUT
erl_csv          370.6      266.1      39832.4    26145.4      TIMEOUT    TIMEOUT

Big integers

JSON/YAML/CSV numbers that don't fit into a 64-bit integer are decoded as Erlang big integers (and big integers are encoded back to their exact decimal representation).

API

encode_integer/1 and decode_integer/1/try_decode_integer/1 expose the same conversion routines directly, independent of JSON/YAML/CSV parsing/encoding:

1> glazer:encode_integer(123456789012345678901234567890).
<<"123456789012345678901234567890">>

2> glazer:decode_integer(<<"123456789012345678901234567890">>).
123456789012345678901234567890

3> glazer:try_decode_integer(<<"not a number">>).
{error, invalid_number_format}

See the module's documentation (src/glazer.erl) for full type specs and details.

Testing

make test

runs the EUnit test suite via rebar3 eunit.

License

MIT License — see LICENSE for details.