erlav
High performance avro encoding/decoding NIF-based library
Build
$ rebar3 compile
$ rebar3 eunit
Usage
SchemaId = erlav_nif:erlav_init(<<"priv/tschema2.avsc">>),
Term = #{
<<"intField">> => 789,
<<"longField">> => 2989898111,
<<"doubleField">> => 11.2345,
<<"floatField">> => 23.12,
<<"boolField">> => true,
<<"stringField">> => <<"asdadasdasdasd3453534dfgdgd123456789">>,
<<"bytesField">> => <<1,99,57,127,0,56>>
},
% Encode Term to avro format
Ret = erlav_nif:erlav_encode(SchemaId, Term),
% Decode data in avro format to erlang term
DecodedTerm = erlav_nif:erlav_decode_fast(SchemaId, Ret)
Negative block-count encoding
Avro allows array/map blocks to be encoded either as a plain positive item
count, or as a negative count followed by the block's byte length (used by
readers to skip a block without decoding it). erlav_decode/
erlav_decode_fast always understand both forms, so bytes produced by other
Avro implementations (e.g. erlavro, which emits the negative form by
default) decode correctly out of the box.
erlav_encode emits the plain positive-count form by default. To make it
emit the negative form instead -- e.g. to match erlavro's byte output, or
for interop with a reader that expects it -- pass
use_negative_block_count when initializing the schema:
SchemaId = erlav_nif:erlav_init(<<"priv/tschema2.avsc">>, [use_negative_block_count]),
% Every array/map field encoded with this SchemaId now uses the negative
% block-count form; decoding is unaffected either way.
Ret = erlav_nif:erlav_encode(SchemaId, Term),
DecodedTerm = erlav_nif:erlav_decode_fast(SchemaId, Ret)
Plain erlav_init/1 (no options) keeps emitting today's positive-count
form, unchanged.
Decoding long strings without copying: decode_sub_binaries
By default erlav_decode_fast/2 copies every decoded string and bytes value
into a new binary. For values of 65 bytes or more the BEAM has to allocate a
reference-counted binary for each copy, which is most of the decode time for
string-heavy messages. Pass decode_sub_binaries to return those values as
sub-binaries of the input instead (no copy, no allocation):
SchemaId = erlav_nif:erlav_init(<<"priv/tschema2.avsc">>, [decode_sub_binaries]),
Decoded = erlav_nif:erlav_decode_fast(SchemaId, Binary)
Measured on array<string> (100 strings per array), time per
erlav_decode_fast/2 call (range over two runs):
| String length | Copy (default) | decode_sub_binaries |
Speedup |
|---|---|---|---|
| 32 B | 4.7-5.2 us | 4.8-5.4 us | none (both copy) |
| 64 B | 6.6-6.8 us | 6.7-7.0 us | none (both copy) |
| 65 B | 16-17 us | 4.0-4.2 us | 3.9-4.2x |
| 136 B | 16.5-18.6 us | ~3.9 us | 4.3-4.7x |
| 500 B | 17.5-19 us | ~4.0 us | 4.4-4.8x |
| 2000 B | 25-26 us | 4.7 us | 5.3-5.7x |
Strings of 65 bytes or more decode about 4-5x faster, and the decode time stays flat as strings get longer. Strings under 65 bytes are unchanged, because they are still copied (a heap binary is cheaper than a sub-binary reference). This is close to a best case, since the benchmark is almost pure string decoding; messages that mix many other field types gain less. The decoded values compare equal either way.
The trade-off: memory. A sub-binary keeps the entire input binary alive for
as long as any decoded value is referenced. If you keep one 136-byte string out
of a 1.4 MB message, the process holds on to the whole 1.4 MB until that string
is dropped. If your code stores slices of decoded messages long-term (a cache,
a long-lived process state), either leave this option off or copy what you keep
with binary:copy/1. (erlavro's decoder behaves like the option: it returns
sub-binaries.) It is off by default for this reason.
Options are part of a schema's identity: the same file initialised with
different options gets a different SchemaId. decode_sub_binaries affects
erlav_decode_fast/2 only; the legacy erlav_decode/2 does not decode strings.
Performance
run erlang shell
erl -pa _build/default/lib/*/ebin
run performance test ( it will generate report at the end )
erlav_perf:all_tests(). %% encode and decode, one table
erlav_perf:all_encode_tests(10000, 50, null).
erlav_perf:all_decode_tests(10000, 50, null).
Every row times erlavro and erlav on the same schema and data and checks that
both produce the same result ( Equal ). Encode rows compare erlavro's
encoder with erlav_encode/2; decode_* rows compare erlavro's decoder with
erlav_decode_fast/2 on binaries produced by erlavro. The CI perf job
(ci/perf_check.sh) runs the whole table and fails if any speedup drops below
a floor or any Equal is false.
Numbers below are one run on one machine ( Linux, GCC 14, 5000 iterations ) and vary a lot between runs and machines; the ratio is what is meaningful.
+------------------------------+---------+--------------+--------------+----------+
| Test | Equal | Erlavro us | Erlav us | Speedup |
+------------------------------+---------+--------------+--------------+----------+
| erlav_perf_tst2 | true | 1051.28 | 190.85 | 5.51x |
| erlav_perf_tst3 | true | 1700.04 | 114.33 | 14.87x |
| map_perf_tst1 | true | 174.90 | 53.39 | 3.28x |
| map_perf_tst2 | true | 168.76 | 43.71 | 3.86x |
| array_int_perf_tst | true | 24.92 | 2.74 | 9.10x |
| array_str_perf_tst | true | 175.31 | 28.86 | 6.07x |
| array_map_perf_tst | true | 451.78 | 94.88 | 4.76x |
| decode_perf_tst2 | true | 555.57 | 93.43 | 5.95x |
| decode_perf_tst3 | true | 1127.38 | 169.36 | 6.66x |
| decode_map_perf_tst1 | true | 60.08 | 17.01 | 3.53x |
| decode_map_perf_tst2 | true | 81.91 | 32.46 | 2.52x |
| decode_array_int_perf_tst | true | 26.73 | 2.22 | 12.03x |
| decode_array_str_perf_tst | true | 42.11 | 23.24 | 1.81x |
| decode_array_map_perf_tst | true | 171.52 | 70.62 | 2.43x |
+------------------------------+---------+--------------+--------------+----------+