ExCodecs

An extensible BEAM-native codec framework for Elixir.

ExCodecs provides a unified API for compression, decompression, hashing, checksums, binary encodings, and future content-addressing codecs — all backed by Rust NIFs for high throughput on the BEAM.

Design Philosophy

ExCodecs is not a compression library. It is a codec framework.

Compression is merely the first codec category. The architecture supports future expansion into hashing, checksums, binary encodings, content addressing, and streaming -- without changing the public API. Every codec implements the ExCodecs.Codec behaviour and registers with the ExCodecs.CodecRegistry at startup, meaning new categories slot in without touching existing code.

The encode/decode naming is category-agnostic: for compression codecs, encoding is compressing and decoding is decompressing; for a future hash codec, encoding would produce a digest and decoding would verify it.

Installation

Add ex_codecs to your list of dependencies in mix.exs:

def deps do
  [
    {:ex_codecs, "~> 0.1.0"}
  ]
end

Then fetch dependencies and compile:

mix deps.get && mix compile

Precompiled NIF binaries are available for macOS (Intel and ARM64), Linux (x86_64 and ARM64, glibc and musl), and Windows (x86_64). They are downloaded automatically from the GitHub releases when you run mix deps.get. If a precompiled artifact is not available for your target, ExCodecs falls back to compiling the Rust NIF from source (requires Rust 1.85+).

Quick Start

# Compress data
{:ok, compressed} = ExCodecs.encode(:zstd, "hello world")
{:ok, original}  = ExCodecs.decode(:zstd, compressed)
original #=> "hello world"

# Compression with options
{:ok, compressed} = ExCodecs.encode(:zstd, my_binary, level: 9)
{:ok, compressed} = ExCodecs.encode(:blosc2, my_binary, cname: :zstd, clevel: 5, shuffle: :byte)

# Discover available codecs
ExCodecs.available_codecs()  #=> [:blosc2, :bzip2, :lz4, :snappy, :zstd]
ExCodecs.supports?(:zstd)    #=> true
ExCodecs.codec_info(:zstd)   #=> {:ok, %ExCodecs.Codec{name: :zstd, category: :compression, ...}}

# Convenience aliases for compression
{:ok, compressed} = ExCodecs.Compression.compress(:lz4, data)
{:ok, original}   = ExCodecs.Compression.decompress(:lz4, compressed)

API Overview

encode/3

{:ok, encoded} = ExCodecs.encode(:zstd, binary, level: 3)

Encodes (compresses) data with the given codec and options. Returns {:ok, binary} on success or {:error, %ExCodecs.Error{}} on failure.

decode/3

{:ok, decoded} = ExCodecs.decode(:zstd, compressed)

Decodes (decompresses) data with the given codec. Returns {:ok, binary} on success or {:error, %ExCodecs.Error{}} on failure.

available_codecs/0

ExCodecs.available_codecs()  #=> [:blosc2, :bzip2, :lz4, :snappy, :zstd]

Returns a sorted list of codec atoms that are both registered and have a loaded native implementation.

supports?/1

ExCodecs.supports?(:zstd)       #=> true
ExCodecs.supports?(:nonexistent) #=> false

Returns true only if the codec is registered and its native NIF is loaded.

codec_info/1

{:ok, info} = ExCodecs.codec_info(:zstd)
info.category       #=> :compression
info.native?        #=> true
info.streaming?     #=> true
info.configurable?  #=> true
info.version        #=> approximate (e.g., "1.5.x")

Returns a structured %ExCodecs.Codec{} struct with metadata, or {:error, :unsupported_codec}.

Supported Codecs

Architecture

ExCodecs is layered as follows:

1. Public API (ExCodecs) -- encode/3, decode/3, available_codecs/0, supports?/1, codec_info/1. All consumers interact with this module.

2. Codec Behaviour (ExCodecs.Codec) -- A behaviour requiring encode/2 and decode/2 callbacks. Each codec module implements this behaviour and optionally exports __codec_info__/0 for registry metadata.

3. Codec Registry (ExCodecs.CodecRegistry) -- An ETS-backed registry populated at application startup. It maps codec atoms to their implementing modules and metadata. Lookups are O(1).

4. Native NIFs (ExCodecs.Native) -- Rustler NIFs providing the actual compression and decompression. Precompiled via rustler_precompiled for cross-platform distribution; falls back to local compilation.

5. Category Modules (ExCodecs.Compression) -- Convenience modules that delegate to the public API with category-specific naming (e.g., compress/decompress).

To add a new codec: implement ExCodecs.Codec, add a native NIF function, register the codec in ExCodecs.Application, and the rest follows automatically.

Error Handling

All public functions return {:ok, result} or {:error, %ExCodecs.Error{}}. Error reasons are atoms:

{:error, error} = ExCodecs.encode(:unknown, "data")
error.reason  #=> :unsupported_codec

{:error, error} = ExCodecs.encode(:zstd, "data", level: 99)
error.reason  #=> :invalid_options

Benchmarking

ExCodecs includes benchmarking utilities via benchee.

# Run all compression benchmarks
mix benchmarks

# Run a specific benchmark file
mix bench compression

Benchmarks are defined in bench/ and run in the :bench environment. Results are saved to bench/results/ (git-ignored).

Development

# Fetch dependencies and compile (includes NIF compilation)
mix deps.get && mix compile

# Run the full test suite
mix test

# Run tests with coverage
mix coveralls

# Run static analysis
mix credo
mix dialyzer

# Format code
mix format

# Lint Rust NIF code
mix rust.lint

# Run Rust NIF tests
mix rust.test

# Generate documentation
mix docs

Requirements

• Elixir 1.17+
• Erlang/OTP 26+
• Rust 1.85+ (only required if precompiled NIFs are unavailable for your platform)

License

Apache License, Version 2.0. See LICENSE for details.