lz4_nif

Erlang NIF wrapper around the LZ4 block-format codec — fast, non-streaming compression and decompression with the standard compress(Binary) / uncompress(Binary, OrigSize) API.

Why

Replaces git-ref-only dependencies on Erlang LZ4 forks. Modern build toolchain (correct OTP 27+ -eval order, macOS -undefined dynamic_lookup, dirty-scheduler dispatch for large inputs), tested against OTP 25-28 in CI, and published to hex.pm.

Install

{deps, [{lz4_nif, "0.1.0"}]}.

Requires a C compiler (cc) on the build host — universally available on systems that already run Erlang.

API

-spec lz4_nif:compress(binary()) ->
    {ok, binary()} | {error, error_reason()}.

-spec lz4_nif:compress(binary(), [option()]) ->
    {ok, binary()} | {error, error_reason()}.

-spec lz4_nif:uncompress(binary(), non_neg_integer()) ->
    {ok, binary()} | {error, error_reason()}.

-type error_reason() :: alloc_failed
                      | badarg
                      | compress_failed
                      | decompress_failed.

The compressed binary does not include the original size; callers track that separately and pass it to uncompress/2. This matches the LZ4 block format convention used by, e.g., the Cassandra/Scylla binary protocol.

The compress/2 options list is currently a placeholder for forward compatibility. No options are interpreted.

1> {ok, C} = lz4_nif:compress(<<"hello, lz4_nif">>).
{ok, <<...>>}
2> {ok, <<"hello, lz4_nif">>} = lz4_nif:uncompress(C, 14).

Behaviour notes

• Dirty CPU scheduler for inputs above 20 KB. Smaller inputs run inline on the calling scheduler; larger inputs are dispatched via enif_schedule_nif so they don't block other processes on the same scheduler. The 20 KB threshold matches the convention used elsewhere in this ecosystem (e.g., torque).
• Output sizing: uncompress/2 requires the caller to pass the exact original size. Wrong size → {error, decompress_failed}. Corrupt input is detected by LZ4_decompress_safe and reported the same way.
• Maximum input size: LZ4_MAX_INPUT_SIZE (≈ 2.1 GB). Larger inputs return {error, badarg}.

Build

rebar3 compile runs c_src/build.sh:

• Resolves ERTS_INCLUDE_DIR via erl -noshell -eval ... -s init stop (option order is correct for OTP 27+ — the bug that broke granderl 0.1.5 is fixed here).
• Compiles c_src/lz4_nif.c + c_src/lz4/lz4.c with -O3 -march=native.
• Outputs priv/lz4_nif.so.

Env vars honored:

License

The Erlang wrapper code (src/, c_src/lz4_nif.c) is MIT. The vendored LZ4 library in c_src/lz4/ is BSD-2-Clause, copyright Yann Collet — see c_src/lz4/LICENSE for the upstream license text.