enif_protobuf — fast Google Protobuf encoder/decoder for Erlang and Elixir (NIF)

enif_protobuf is a Google Protobuf encoder and decoder for Erlang/OTP, implemented as a NIF in C. It is a drop-in replacement for the encode_msg/1 and decode_msg/2 functions that gpb generates from .proto files: same record tuples, same wire format, same results — but serialization runs in C instead of Erlang, with no C++ libprotobuf dependency.

Use gpb's protoc-erl to generate the Erlang modules as usual, load the schema once with enif_protobuf:load_cache(Module:get_msg_defs()), then call enif_protobuf:encode/1 and enif_protobuf:decode/2. The same calls work from Elixir, since the API is plain Erlang.

Contents

• Why use enif_protobuf
• Installation
• Quick start
• API reference
• Supported protobuf features
• Options
• Performance
• Comparison with alternatives
• Using enif_protobuf with AI coding assistants
• FAQ
• Development

Why use enif_protobuf

• Roughly 1.8–2.6× faster than pure-Erlang gpb, and roughly 2.6–9× faster than gpb's libprotobuf NIF, on the standard Google benchmark messages (see numbers).
• Byte-compatible with gpb. The test suite asserts gpb:encode_msg/1 =:= enif_protobuf:encode/1 for the covered messages, so existing consumers keep working unchanged.
• No runtime dependencies beyond OTP. One small NIF (priv/enif_protobuf.so), no libprotobuf, no protoc at runtime, no per-module native build.
• Full protobuf feature set — proto2 and proto3, every scalar type, enums (including aliases and negative values), repeated and packed fields, nested messages, oneofs, maps, groups, unknown-field passthrough.
• No regeneration step for existing gpb users. The schema comes from Module:get_msg_defs(), which your generated modules already export.

Installation

Add it from Hex in rebar.config:

{deps, [
    {enif_protobuf, "~> 1.0"}
]}.

Or track master directly:

{deps, [
    {enif_protobuf, {git, "https://github.com/jinganix/enif_protobuf.git", {branch, "master"}}}
]}.

The NIF is compiled by a pre_hooks entry during rebar3 compile, so a working C toolchain (cc/clang, make) is required on the build machine (the Hex package ships sources, not a prebuilt .so). Erlang/OTP 28 or newer is required; the CI matrix covers OTP 28 and 29 on Linux and macOS. API docs are published at hexdocs.pm/enif_protobuf.

Build

make            # compile (rebar3 + C NIF)
make tests      # EUnit
make c-tests    # C unit tests
make c-format-check  # verify C style (clang-format >= 19)
make c-format        # apply clang-format to c_src/ and c_src_tests/

With rebar3 directly:

rebar3 compile
rebar3 eunit

Quick start

Given x.proto:

message Person {
  required string name = 1;
  required int32 id = 2;
  optional string email = 3;
}

Generate and compile Erlang with gpb:

export GPB_PATH=/path/to/gpb
${GPB_PATH}/bin/protoc-erl -I. x.proto
erlc -I${GPB_PATH}/include x.erl

With rebar3, compiled beams live under _build/default/lib/.../ebin.

Load the schema cache once, then encode and decode:

export EPB_EBIN_PATH=/path/to/enif_protobuf/_build/default/lib/enif_protobuf/ebin
erl -pa ${EPB_EBIN_PATH} -pa .

1> rr("x.hrl").
['Person']
2> Msg = #'Person'{name = "abc def", id = 345, email = "a@example.com"}.
3> gpb_bin = x:encode_msg(Msg).
4> enif_protobuf:load_cache(x:get_msg_defs()).
ok
5> epb_bin = enif_protobuf:encode(Msg).
6> gpb_bin =:= epb_bin.
true
7> enif_protobuf:decode(epb_bin, 'Person').
#'Person'{name = <<"abc def">>, id = 345, email = <<"a@example.com">>}

Convenience wrappers that load the cache on each call:

enif_protobuf:encode_msg(Msg, x:get_msg_defs()).
enif_protobuf:decode_msg(Bin, 'Person', x:get_msg_defs()).

API reference

Calling encode/1 or decode/2 before load_cache/1 raises cache_not_exists. All functions are safe to call concurrently from many processes: the schema cache is shared and read-locked, and each scheduler gets its own scratch buffers.

Supported protobuf features

string fields decode to binaries by default; use {string_as_list, true} to get lists instead. Non-Latin-1 text requires {with_utf8, true} when encoding (see below).

Options

Call enif_protobuf:set_opts/1 before load_cache/1:

%% UTF-8 strings require with_utf8
enif_protobuf:set_opts([{with_utf8, true}]).
enif_protobuf:encode(#'Person'{name = "你好", id = 345, email = "a@example.com"}).

Other API: purge_cache/0 clears the loaded schema; debug_term/1 prints internal term layout.

Performance

Comparison of enif_protobuf (NIF), gpb (pure Erlang), and gpb NIF (libprotobuf) on the standard Google benchmark messages (228 and 84584 bytes on the wire). Regenerate on your machine:

asdf install   # uses .tool-versions (erlang 29.0.1)
make benchmark-update-readme

Benchmarks always use the Erlang version from .tool-versions. See benchmarks/README.md for all targets (maps, d.proto, gpb NIF, etc.).

Last run: 2026-11-02 16:09:07 UTC (standard Google benchmark messages; Erlang 29.0.1 per .tool-versions; machine Apple M4 Pro)

Higher is better. Throughput in MB/s on serialized wire size.

Comparison with alternatives

The NIF path adds a native build step; if you cannot ship a .so (restricted CI, no compiler), stay with pure gpb — the API is the same, so switching later is a one-line change.

Using enif_protobuf with AI coding assistants

This repo ships machine-readable context so LLM assistants (Claude Code, Codex, Cursor, Copilot, pi, …) produce working code on the first try:

• AGENTS.md — repo rules, architecture map, commands, and the API pitfalls that cause most wrong answers.
• llms.txt — compact, llms.txt-spec summary of the project for retrieval-based assistants.

Rules an assistant should follow when writing code against this library:

1. Generate modules with gpb protoc-erl; do not hand-write record definitions.
2. Call enif_protobuf:load_cache(Module:get_msg_defs()) once at startup (or use encode_msg/2 / decode_msg/3, which load it per call).
3. Preserve the gpb record shape: field order and types must match Module:get_msg_defs().
4. Strings: pass lists on encode, expect binaries on decode unless {string_as_list, true} is set; non-Latin-1 text needs {with_utf8, true} before load_cache/1.
5. Decode needs the message name atom, e.g. enif_protobuf:decode(Bin, 'Person').
6. Do not invent options: only string_as_list and with_utf8 exist.

Example prompt that works well:

Convert this module to use enif_protobuf instead of gpb. Keep the record definitions and generated modules, load the schema once in start/2, and keep the wire format unchanged.

FAQ

Is enif_protobuf faster than gpb?

Yes, in this repo's benchmarks it is ~1.8–2.6× faster than pure-Erlang gpb and several times faster than gpb's libprotobuf NIF. Numbers and hardware are in the benchmark table; rerun make benchmark-update-readme on your own machine before relying on them.

Do I need to regenerate my .proto files or change my records?

No. Keep gpb's protoc-erl output and record definitions; only the encode/decode calls change.

Does it produce the same bytes as gpb?

Yes for the messages covered by the test suite (the tests assert gpb_bin =:= epb_bin). Both follow the protobuf wire format, so any conforming decoder reads either.

Can I use it from Elixir?

Yes. It is an Erlang library with no non-Erlang API: add it to mix.exs as a dependency ({:enif_protobuf, "~> 1.0"}) and call :enif_protobuf.decode(bin, :'Person') instead of enif_protobuf:decode(Bin, 'Person'). Generate the schema module with gpb's protoc-erl (or protoc-gen-elixir) and load it once with :enif_protobuf.load_cache(X.get_msg_defs()).

Why does encode/1 or decode/2 raise cache_not_exists?

No schema is loaded yet (or purge_cache/0 was called). Call enif_protobuf:load_cache/1 first, or use encode_msg/2 / decode_msg/3.

Why do I get badarg from set_opts/1?

Only {string_as_list, boolean()} and {with_utf8, boolean()} are accepted, and the argument must be a proper list of such tuples.

How do I encode non-ASCII (UTF-8) strings?

enif_protobuf:set_opts([{with_utf8, true}]) before load_cache/1. Without it, encoding raises on code points outside Latin-1.

Are strings returned as binaries or lists?

Binaries by default. Set {string_as_list, true} for lists.

Can several processes or applications use it at once?

Encoding and decoding are safe from any number of processes. The schema cache is global to the node, so two applications with different .proto schemas must not share one VM — give each its own node, or coordinate the single load_cache/1 call.

Does it run on dirty schedulers?

No; calls execute on normal schedulers and hold a NIF read lock for the duration of the call. Keep individual messages reasonably sized, or shard very large payload work across pools.

Why does memory stay high after encoding one huge message?

The per-scheduler encode buffer grows geometrically to fit the largest message encoded and is not shrunk afterwards; it is released when the NIF is unloaded (VM exit). This is a deliberate trade-off for the fast path.

Does it work on Windows?

rebar.config has a win32 make hook, but CI only covers Linux and macOS.

Development

make              # rebar3 compile + build the NIF
make tests        # EUnit (includes gpb compatibility and PropEr round-trip tests)
make c-tests      # C unit tests
make c-coverage-check
make c-format     # clang-format >= 19 applied to c_src/ and c_src_tests/

Layout: src/enif_protobuf.erl (public API + NIF stubs) → c_src/enif_protobuf.c (NIF entry points, schema parsing, cache), c_src/ep_encoder.c / c_src/ep_decoder.c (wire codec), test/ (EUnit, gpb parity, PropEr fuzzing), benchmarks/ (proto-bench suite).

See AGENTS.md for the invariants that must hold in this codebase — an uncaught C exception or a bad term handled in the NIF kills the whole VM.

Releasing

Two ways, same result: the source package plus edoc docs land on hex.pm/packages/enif_protobuf and hexdocs.pm/enif_protobuf.

From CI (preferred). Bump {vsn, ...} in src/enif_protobuf.app.src, commit, then push a matching tag. .github/workflows/release.yml runs the tests, publishes to Hex and creates the GitHub release. It refuses to publish when the tag and the .app.src version disagree (v1.0.0 must be {vsn, "1.0.0"}). workflow_dispatch publishes the current master without tagging.

gh secret set HEX_API_KEY                    # once: key from https://hex.pm/dashboard/keys
git tag v1.0.1 && git push origin v1.0.1     # releases are cut by pushing a tag

From your machine, the way jiffy does it:

rebar3 hex user auth       # once; stores the API key under ~/.config/rebar3/hex/
make release               # = rebar3 hex publish, uses the {vsn, ...} from app.src

Either way the package contains sources only ({files, ...} in the .app.src): users compile the NIF with their own C toolchain, so there are no per-platform binaries to publish. Check the tarball before publishing with ./rebar3 hex build (output in _build/default/lib/enif_protobuf/hex/).

License

Apache-2.0.