ExOpenzl

Elixir NIF bindings for OpenZL, Meta's format-aware compression framework.

OpenZL extends zstd with typed and columnar compression, enabling significantly better ratios on structured data like timestamps, enums, and variable-length strings. It also includes SDDL (Structured Data Description Language) for defining custom format-aware compression graphs.

Features

• One-shot compression — compress/decompress binaries with a single call
• Reusable contexts — amortize allocation cost across many operations
• Configurable compression levels — levels 1-19 (same as zstd)
• Typed compression — numeric, struct, and string-aware encoding
  • Numeric: delta coding for monotonic sequences (timestamps, counters)
  • Struct: fixed-width record encoding
  • String: variable-length with packed length arrays
• Multi-typed frames — pack multiple typed columns into a single compressed frame
• Frame introspection — query metadata without decompressing
• SDDL compressor — compile and apply format-aware compression graphs

Prerequisites

• Erlang/OTP 26+
• Elixir 1.19+
• CMake 3.14+
• C++17 compiler (clang or gcc)
• Git (for fetching the OpenZL submodule)

Installation

Add ex_openzl to your dependencies in mix.exs:

def deps do
  [
    {:ex_openzl, "~> 0.4"}
  ]
end

The NIF is compiled automatically via elixir_make. The first build will compile the OpenZL C library from source (takes ~1 minute).

Precompiled binaries

Precompiled NIF binaries are available for the following targets:

• x86_64-linux-gnu
• aarch64-linux-gnu
• x86_64-apple-darwin
• aarch64-apple-darwin

If a precompiled binary is available for your platform, mix compile will download it automatically — no C++ toolchain required.

Usage

Basic compression

data = "hello world, this is a test of OpenZL compression"

{:ok, compressed} = ExOpenzl.compress(data)
{:ok, ^data} = ExOpenzl.decompress(compressed)

One-shot compression creates a fresh native context. For repeated operations, reuse the per-process contexts shown below to amortize native setup cost.

Decompression rejects frames that declare more than 256 MiB of output before allocating that output. Trusted larger frames can use an explicit byte limit:

{:ok, decompressed} = ExOpenzl.decompress(compressed, 1024 * 1024 * 1024)

Reusable contexts

{:ok, cctx} = ExOpenzl.create_compression_context()
:ok = ExOpenzl.set_compression_level(cctx, 9)

{:ok, compressed} = ExOpenzl.compress(cctx, data)

{:ok, dctx} = ExOpenzl.create_decompression_context()
{:ok, ^data} = ExOpenzl.decompress(dctx, compressed)

Typed columnar compression

Pack structured data into typed columns for better compression ratios:

{:ok, cctx} = ExOpenzl.create_compression_context()

# Timestamps: packed u64 little-endian (delta coding)
timestamps = <<1000::little-64, 1001::little-64, 1002::little-64>>

# Levels: u8 enum values (entropy coding)
levels = <<0, 1, 1>>

# Messages: concatenated strings + packed little-endian u32 lengths
messages = "helloworld!!"
msg_lengths = <<5::little-32, 5::little-32, 2::little-32>>

{:ok, compressed} = ExOpenzl.compress_multi_typed(cctx, [
  {:numeric, timestamps, 8},
  {:numeric, levels, 1},
  {:string, messages, msg_lengths}
])

# Decompress
{:ok, dctx} = ExOpenzl.create_decompression_context()
{:ok, [ts_info, lv_info, msg_info]} = ExOpenzl.decompress_multi_typed(dctx, compressed)

Small multi-output frames

Versions before 0.4.10 may return {:error, "Destination capacity too small..."} from compress_multi_typed/2 for very small batches split across multiple typed outputs. If you see this error with valid inputs, upgrade to 0.4.10 or later.

Frame introspection

{:ok, info} = ExOpenzl.frame_info(compressed)
# => %{format_version: 1, num_outputs: 3, outputs: [...]}

SDDL format-aware compression

{:ok, compiled} = ExOpenzl.sddl_compile("u32 timestamp; u8 level;")
{:ok, compressor} = ExOpenzl.create_sddl_compressor(compiled)

{:ok, cctx} = ExOpenzl.create_compression_context()
:ok = ExOpenzl.set_compressor(cctx, compressor)

# Now compress/2 uses the format-aware graph
{:ok, compressed} = ExOpenzl.compress(cctx, data)

Thread safety

Compression and decompression contexts are not thread-safe. Each context should be used by a single Erlang/Elixir process at a time. If you need to compress or decompress from multiple concurrent processes, create a separate context per process.

Hardware acceleration

On x86-64, OpenZL uses BMI2 assembly for Huffman coding, SSE2/AVX2 SIMD for match finding, and BMI2 intrinsics for varint encoding. On other architectures (including Apple Silicon), it falls back to portable C.

License

MIT - see LICENSE.

OpenZL itself is licensed under the BSD License by Meta Platforms, Inc.