LeanLmdb

Fast, safe, and ergonomic LMDB storage for Elixir, powered by Rust.

Installation

Add the Hex package to mix.exs:

def deps do
  [{:lean_lmdb, "~> 0.3.0"}]
end

Quickstart

All names, keys, values, CAS payloads, and continuation tokens are binaries; there is no Elixir-term serialization.

path =
  Path.join(
    System.tmp_dir!(),
    "lean_lmdb_example_#{System.unique_integer([:positive, :monotonic])}"
  )
{:ok, env} = LeanLmdb.open(path, map_size: 64 * 1024 * 1024)
{:ok, users} = LeanLmdb.create_database(env, "users")
{:ok, cache} = LeanLmdb.create_database(env, "cache")

:not_found = LeanLmdb.get(users, "user:1")
:ok = LeanLmdb.put(users, "user:1", <<0, 255>>)
{:ok, <<0, 255>>} = LeanLmdb.get(users, "user:1")
:ok = LeanLmdb.delete(users, "user:1")

:ok = LeanLmdb.write_batch(env, [
  {:put, users, "user:1", "Ada"},
  {:put, users, "user:2", "Grace"},
  {:delete, cache, "stale"}
])

:ok = LeanLmdb.compare_exchange(users, "user:3", :missing, {:put, "Lin"})
{:conflict, "Lin"} =
  LeanLmdb.compare_exchange(users, "user:3", :missing, {:put, "Other"})

{:ok, first, token} = LeanLmdb.scan(users, prefix: "user:", limit: 2)
{:ok, second, :done} =
  LeanLmdb.scan(users, prefix: "user:", limit: 2, continuation: token)
true = first ++ second == [{"user:1", "Ada"}, {"user:2", "Grace"}, {"user:3", "Lin"}]

Use a unique path in real tests and remove it only after all handles are unreachable. There is no force-close API.

Public result contracts

See the generated API documentation for each function's validation rules and stable reasons. Native failures return atoms, never platform error prose.

Lifecycle and transaction containment

Within one BEAM OS process, compatible opens of the same canonical path share native environment state. Opaque Environment handles and immutable Database handles may remain live across calls; a database handle retains the shared environment state. No heed transaction or cursor is an Elixir resource. Every transaction/cursor starts and ends in one dirty-I/O NIF call.

Reads copy values into BEAM-owned binaries before ending the transaction. Scans also copy every key, value, and token. LeanLmdb is therefore intentionally not an end-to-end zero-copy API. NIF hot upgrade while native handles exist is not supported; restart the BEAM when replacing the native library.

Environment defaults, map sizing, and durability

open/2 defaults to a 64 MiB map, 16 named databases, 126 readers, writable access, create: true, durability: :sync, fixed_map: false, and read_ahead: true. map_size is fixed, page-size aligned, and limited to 1 MiB..1 TiB. max_dbs is 1..1024 and max_readers is 1..4096. LeanLmdb does not resize maps or retry failed operations. Size the map and reader/database limits before opening.

Durability is selected per environment:

# Fully synchronized LMDB commits (default).
{:ok, durable} = LeanLmdb.open(path, durability: :sync)

# Preserve ACI, but a system crash may undo the latest transaction.
{:ok, metadata_deferred} = LeanLmdb.open(path, durability: :no_meta_sync)

# Do not flush on each commit; periodically force a durable checkpoint.
{:ok, derived} = LeanLmdb.open(path, durability: :no_sync)
:ok = LeanLmdb.sync(derived)

durability: :no_sync can lose recent commits and can corrupt the database on a system crash when the filesystem does not preserve write order. This mode is appropriate only when the store is disposable and can be rebuilt from an authoritative source. sync/1 calls forced mdb_env_sync; applications can run it from a supervised timer to implement delayed group durability.

fixed_map: true exposes experimental MDB_FIXEDMAP. The recorded virtual address must be available in every process opening the environment; open can fail under ASLR/address-space conflicts. WRITE_MAP, MAP_ASYNC, and NO_LOCK remain unavailable. Common capacity/operational results include :map_full, :map_resized, :readers_full, :transaction_full, :out_of_memory, and :io_error.

read_ahead: false requests LMDB's MDB_NORDAHEAD hint. It can improve random reads when the database is larger than available RAM and the page cache is under pressure, but can hurt sequential range/prefix scans. It has no effect on Windows. Benchmark the actual working set before disabling the default:

{:ok, random_access} = LeanLmdb.open(path, read_ahead: false)

CRUD, atomic mutation, and single-writer behavior

Keys are non-empty and at most LMDB's runtime maximum (currently bounded by 511 bytes here). Values may be empty and can contain arbitrary bytes. put/3 overwrites; delete/2 is idempotent. A failed write aborts its transaction.

Batches contain at most 10,000 operations and 64 MiB of aggregate key/value input. All database handles must belong to the supplied environment. The full batch is validated before one write transaction, applied in list order, and committed once. CAS distinguishes :missing from <<>> and returns the exact copied current value on conflict.

LMDB allows concurrent readers but exactly one active writer per environment, including across OS processes. A dirty-I/O call can wait for that writer; short batches improve useful work per serialized commit without changing durability.

Opt-in value codecs and string keys

Elixir strings are binaries, so readable keys work without a key codec:

LeanLmdb.get(database, "user:1")
LeanLmdb.scan(database, prefix: "user:")

Value serialization is opt-in and remains outside the NIF:

alias LeanLmdb.Codecs.ErlangTerm

{:ok, raw} = LeanLmdb.create_database(environment, "users")
{:ok, users} = LeanLmdb.with_value_codec(raw, ErlangTerm)

:ok = LeanLmdb.put(users, "user:1", %{name: "Alice", roles: [:admin]})
{:ok, %{name: "Alice", roles: [:admin]}} = LeanLmdb.get(users, "user:1")

Codec values carry a bounded versioned envelope. Opening raw legacy bytes through a codec fails explicitly; it never guesses or deserializes them. raw_database/1 is the intentional diagnostics/migration escape hatch. The built-in term codec uses deterministic external-term encoding, safe decoding, an 8 MiB default encoded limit, and rejects compressed external terms, PIDs, ports, references, functions, and oversized payloads. The limit is not an exact BEAM heap bound; size it below the process memory budget.

Codec scans decode values while keys/tokens remain raw. Complete batches are encoded before native mutation, so an encoding failure commits nothing. Logical CAS is supported only for deterministic codecs because LMDB atomically compares the encoded envelope bytes:

:ok = LeanLmdb.compare_exchange(users, "user:1", %{name: "Alice", roles: [:admin]},
  {:put, %{name: "Alice", roles: [:admin, :operator]}}
)

A codec mismatch returns {:error, :codec_mismatch} and malformed legacy bytes return {:error, :codec_decode_failed}; LeanLmdb never falls back to raw data or selects a module from the stored codec ID. To migrate a legacy database, retain its raw handle, page through raw values with bounded scans, decode each value using explicit application migration code, and write it through the selected codec wrapper. Overwrite only after decoding succeeds, checkpoint progress in the authoritative store, and keep the old decoder until every supported stored version is migrated. Wrapping a database performs no migration by itself.

Custom codecs implement LeanLmdb.ValueCodec and may serialize, compress, or encrypt payloads. Compression codecs must enforce their own decoded-size bounds; scan max_bytes counts encoded bytes copied by the NIF.

Supervised group commit

LeanLmdb.BatchWriter is an optional OTP process that combines concurrent submissions into bounded FIFO transactions:

{:ok, writer} =
  LeanLmdb.BatchWriter.start_link(
    environment: environment,
    max_batch_operations: 1_000,
    max_batch_bytes: 8 * 1024 * 1024,
    max_wait_ms: 5
  )

:ok = LeanLmdb.BatchWriter.put(writer, users, "user:1", %{name: "Alice"})
:ok = LeanLmdb.BatchWriter.flush(writer)

A successful submission means its LMDB transaction committed; it is never only an in-memory acceptance. The queue and each batch are bounded by operation count and encoded bytes, with :batch_writer_overloaded returned at capacity. Codec encoding occurs in the producer before queue admission. status/2 reports sanitized queue high-water, dispatch-trigger, and commit counters.

flush/2 is a commit barrier. For :no_sync and :no_meta_sync environments, sync/2 drains prior work and then forces an environment checkpoint. A caller timeout does not cancel already accepted work. stop/2 stops acceptance, drains, and terminates normally. Under a supervisor the writer uses restart: :transient, so unknown worker exits restart it while a normal stop does not. Forced shutdown does not promise to drain volatile queue contents.

Commit workers are monitored. A normal atomic LMDB error is returned to every waiter represented by that batch. If a worker exits before its matching result is accepted, the commit may already have happened: in-flight callers receive :batch_writer_unknown_outcome, queued callers fail, no operation is retried, and the writer terminates for a clean supervisor restart. Reconcile an unknown outcome against the authoritative event store. The writer is opt-in and is never started automatically by the application.

Bounded scans and tokens

scan/2 is forward-only in raw binary-key order. Choose prefix: binary or a range using from, to, from_inclusive (default true), and to_inclusive (default false). Do not mix prefix and range options. limit defaults to 100 and is 1..10,000. max_bytes defaults to 1 MiB and is 1..64 MiB, counting key plus value bytes. If the first eligible row is larger than the selected budget, the call returns {:error, :row_too_large}; page output never exceeds the byte or row bound. Because 64 MiB is the maximum budget, a key/value row larger than 64 MiB cannot be traversed by scan/2; use smaller/chunked values or a different store design.

A token is an opaque, versioned, process-local binary with a keyed integrity tag, not a cryptographic credential. It contains no pointer or cursor. It binds to the native environment resource ID, database name, and original bounds; limit and max_bytes may change. Modified/mismatched tokens return :invalid_continuation; tokens from a released environment return :stale_continuation. Tokens do not survive a BEAM restart.

Each page has a fresh read transaction, so pages do not share one snapshot. Resume is exclusive after the last emitted key. Writes between pages can add, change, or remove later rows; insertions at or before the last emitted key will not appear. An unchanged database traverses exactly once without duplicates.

Multi-process operation and stale readers

Each BEAM OS process has its own registry and heed handle. LMDB coordinates via its data and lock files. Configure all processes with the same map/database/ reader limits; first-opener and process-local details otherwise can produce :map_resized or incompatible behavior. A commit becomes visible to another process when that process starts a later transaction.

An abrupt writer death cannot publish a partial LMDB transaction, but recovery is not corruption repair. A killed reader can leave a stale lock-table slot, causing file growth or eventually :readers_full. clear_stale_readers/1 uses LMDB's reader check to remove dead-process slots only and is safe to repeat. Never delete the lock file while any process has the environment open.

The deterministic integration suite uses independent BEAM processes, explicit file barriers, bounded polling, and Unix hard kills; it does not infer recovery from sleeps.

Filesystem and operational caveats

Use only a stable local filesystem. NFS, SMB, distributed/network mounts, and moving, replacing, truncating, or externally editing open environment files are unsupported. Available disk and virtual address space and LMDB/platform limits still apply. Plan and test backups and recovery according to your durability requirements. LeanLmdb has no built-in backup, replication, corruption repair, distributed consensus, automatic resize, or native hot-upgrade protocol.

The API fits workloads where data is binary, bounded scans are sufficient, and serialized short writes are acceptable. It is a poor fit when values need streaming/chunking, remote storage is required, or sustained parallel write transactions are required.

Benchmarking

A comprehensive Benchee suite covers point reads, synchronized mutations, atomic batch sizes, bounded and complete scans, mixed traffic, parallel contention, fresh projection rebuilds, and multiple LMDB map/reader/database configurations. It produces p50/p95/p99 statistics, raw JSON, a reproducibility manifest, and interactive HTML charts.

mix deps.get
mix run bench/run.exs -- --profile quick
mix run bench/run.exs -- --configs all --workloads reads --parallel 8

See the benchmarking guide for profiles, all options, workload units, chart interpretation, and warm/cold-cache methodology. Generated reports are written to bench/output/ and are not performance gates.

Source development

Install the versions above, fetch both dependency sets once, then run the single quality gate:

mix deps.get
cargo +1.91.0 fetch --manifest-path native/lean_lmdb_nif/Cargo.toml --locked
bin/qa_check.sh

After dependencies exist, the gate globally forces source fallback, forces Hex and Cargo offline, builds docs with warnings as errors, runs Elixir tests twice, runs all-feature Rust lint and tests, and performs a release build without test-support. It also checks an optional checksum manifest is packaged unchanged, compiles the unpacked package from its included Rust source, then runs its package-built NIF through native_version/0 and isolated CRUD. Every Mix/Cargo command has a portable per-command deadline (900 seconds by default, configurable with QA_COMMAND_TIMEOUT_SECONDS).

Maintainer release flow

Precompiled publication is explicit and draft-first. The workflow never runs on a tag push, so it cannot recurse. For a version already set in mix.exs and the native Cargo.toml:

git push origin main
gh workflow run precompiled-release.yml --ref main -f publish=true
gh run list --workflow precompiled-release.yml --limit 1
gh run watch RUN_ID --exit-status

version=$(bin/project_version.sh)
gh release view "v$version" --json tagName,targetCommitish,isDraft,isPrerelease,assets
mix rustler_precompiled.download LeanLmdb.Native --all --print

The matrix uses target-aware Cargo caches. Every archive is directly loaded on a matching runtime before aggregation; Linux musl uses Alpine. The aggregator accepts exactly seven expected archives, creates a draft from the dispatched commit, verifies GitHub's SHA-256 asset digests, then publishes. A failure after draft creation removes only the draft and tag created by that run.

The generated checksum-Elixir.LeanLmdb.Native.exs is mandatory for the normal precompiled path. Review and commit it, rerun bin/qa_check.sh, then push and wait for the CI precompiled-consumer matrix. Those jobs compile the unpacked Hex package from clean downstream projects with cargo and rustc replaced by failing shims. Use gh run cancel RUN_ID immediately for an obsolete or known- bad run, inspect failures with gh run view RUN_ID --log-failed, and retry only after committing the fix.