ash_vault_rustler

A Rust NIF providing an AshVault key cache and an AES-256-GCM cipher that hold key material outside the BEAM heap, so that eviction can actually zero it.

Optional. The parent ash_vault has no Rust dependency and never will.

def deps do
  [
    {:ash_vault, "~> 0.2.0"},
    {:ash_vault_rustler, "~> 0.2.0"}
  ]
end
use AshVault.Vault,
  key_provider: MyApp.Keys,
  cache: [backend: AshVaultRustler.KeyCache, ttl: :timer.seconds(30)]

What this buys

The BEAM cannot zero memory, and that makes crypto-erasure eventual

AshVault's whole reason to exist is one sentence: destroying a customer's keys makes their historical ciphertext permanently undecryptable. A key cache quietly weakens that. "Evicting" a key from ETS drops a reference; the bytes survive in however many process heaps read them, for however long the garbage collector takes, and there is no BEAM API that overwrites them. Every BEAM crypto library shares this. Cloak does not attempt it — zero hits for zero/wipe/scrub across its lib/, and it keeps every key in a named :protected ETS table for the life of the process.

AshVaultRustler.KeyCache fixes exactly that, and only that:

AshVault.KeyCaches.ETS AshVaultRustler.KeyCache
Eviction is synchronous yes yes
Eviction is observable (no later read can see it) yes yes
Generation fence against a racing repopulate yes yes
Eviction overwrites the bytes no yes
Key pages mlocked out of swap no yes (best effort)
Needs a Rust toolchain or a precompiled artifact no yes

Each cached key lives in a page-aligned allocation the crate owns, mlocked into RAM, and Dropped with a volatile overwrite. evict_scope/1 returns only after those writes have happened.

Opaque key handles

AshVaultRustler.KeyProviders.Opaque wraps any provider so it returns %AshVault.Key{} handles instead of binaries, and AshVaultRustler.Cipher runs the AEAD against a handle. The key never becomes an Elixir term during an encrypt or a decrypt.

A cipher you can drop in without stranding a row

AshVaultRustler.Cipher uses the same id (:aes_256_gcm_v1), the same 12-byte nonce, the same 16-byte tag and the same AAD as AshVault.Ciphers.AES.GCM. A value written by either decrypts with the other; the test suite asserts both directions across sizes from 0 bytes to 200 KB.


What this does not buy

Be precise about this, because the temptation to overclaim here is enormous.

The level-1 cache does not eliminate the per-operation copy. AshVault.KeyProviders.Cached.get_key/3 returns a binary to the BEAM. A copy of the key lands on an Elixir process heap for every single encrypt and decrypt, and that copy is the garbage collector's business, not ours. This package bounds the lifetime of the authoritative copy. Eliminating the per-operation one needs the opaque-handle path, which is a separate opt-in.

Even the opaque path has a boundary. The wrapped provider returns a binary — that is its contract — so the key bytes pass through the BEAM once, on the way into the handle. What goes away is the repeat: with a cache in front, key bytes exist as an Elixir term once per cache miss rather than once per field read.

The TTL is the erasure SLA, but only for erasures that do not go through the vault. AshVault.KeyProviders.Cached.destroy/2 evicts synchronously on every connected node before it returns, and fails if any node does not acknowledge — so an erasure performed through the vault is immediate. An erasure performed out of band is not: a mix ash_vault.destroy_keys in an unclustered node, an operator deleting a transit key in OpenBao by hand, another service revoking it. None of those call evict_scope/1, and the cache serves the erased scope until the entry expires. That TTL is the number you can write in a DPA, and it defaults to 30 seconds.

mlock is best effort, and it is charged per page. Each cached key gets its own page, because mlock/munlock act on whole pages and locking a 32-byte buffer inside a shared page would unlock whatever else lives there when it is freed. So:

A 32-byte key costs 4 KiB of locked memory, and :max_bytes counts key bytes, not pages. The defaults (max_entries: 1_024, max_bytes: 1_048_576) can lock up to 4 MiB while the reported byte total reads 32 KiB. With this backend, size :max_entries against RLIMIT_MEMLOCK, not against :max_bytes.

A default container ulimit -l is often 64 KiB — sixteen pages — so locking starts failing after roughly sixteen cached keys on a stock host. That is not fatal: the failure is logged once, loudly, after the first write that observes it, and caching continues, because a key that might reach swap beats a node that refuses to decrypt. Check AshVaultRustler.mlock_status/0 in production and alert on a non-zero :failed.

Nothing here defends against a compromised application. An attacker with code execution can ask the cache for a key exactly as your app does. See the parent's threat model.


The rule that matters most

Never cache the absence of a tombstone.

{:error, :destroyed} is cached forever: destruction is monotonic, so a cached tombstone can only be right, and caching it makes reads fail closed even when the provider is unreachable. The absence of a tombstone is never cached; nor is {:error, :not_found}, nor a transport failure.

Caching {:ok, key} is, strictly, an implicit cache of "not destroyed", and pretending otherwise would be the fifth fail-open tombstone read in this project's history. Two mechanisms contain it: synchronous cluster-wide eviction on the destroy path, and a short TTL as the floor for everything else. The policy lives in AshVault.KeyProviders.Cached — in Elixir, where it can be read and reviewed — and the Rust crate holds no opinion about tombstones at all.


Configuration

The cipher: set both :cipher and :ciphers

This is the easiest thing to get wrong, and it is a consequence of the wire compatibility being real:

# governs ENCRYPTION
use AshVault.Vault, key_provider: MyApp.Keys, cipher: AshVaultRustler.Cipher

# governs DECRYPTION — required
config :ash_vault, :ciphers, %{"aes_256_gcm_v1" => AshVaultRustler.Cipher}

AshVault.Vault.Runtime.decrypt!/3 resolves the cipher from the envelope, through AshVault.Cipher.registry/0, not from the vault's :cipher option — deliberately, so that values keep decrypting after the default cipher changes. Since this cipher shares the built-in id, a vault configured with only :cipher will encrypt in Rust and decrypt in Elixir. Harmless with a binary key; impossible with an opaque handle, which is where you find out.

The cache

use AshVault.Vault,
  key_provider: MyApp.Keys,
  cache: [
    backend: AshVaultRustler.KeyCache,
    ttl: :timer.seconds(30),          # current key; also the erasure SLA
    historical_ttl: :timer.seconds(30),
    max_entries: 1_024,
    max_bytes: 1_048_576,
    cluster: true,                    # evict on every connected node, fail if any doesn't ack
    evict_timeout: 5_000
  ]

Then supervise the generated provider — it owns the cache:

children = [MyApp.Vault.CachedKeyProvider]

If it is not started, every read goes to the provider. Slower, and correct.

Opaque keys

defmodule MyApp.OpaqueKeys do
  use AshVaultRustler.KeyProviders.Opaque, provider: AshVault.KeyProviders.OpenBao
end

use AshVault.Vault, key_provider: MyApp.OpaqueKeys, cipher: AshVaultRustler.Cipher
config :ash_vault, :ciphers, %{"aes_256_gcm_v1" => AshVaultRustler.Cipher}

A cipher that cannot take a handle raises AshVault.Errors.OpaqueKeyUnsupported naming both modules. There is deliberately no fallback that unwraps the handle — that would put the key back on the BEAM heap, silently, which is the one thing the handle prevents.


Scheduler discipline, and the numbers behind it

Measured on this host (AES-NI, mix run bench/scheduler_bench.exs):

plaintext normal scheduler dirty CPU scheduler OTP :crypto
16 B 1.20 µs 1.73 µs 1.90 µs
64 B 0.68 µs — —
1 KiB 1.70 µs 3.04 µs 2.72 µs
4 KiB 3.92 µs — —
16 KiB 13.6 µs — —
64 KiB 46.4 µs 80.9 µs 42.0 µs
256 KiB 319 µs — —
1 MiB 1529 µs 1055 µs 715 µs
4 MiB 5958 µs 5392 µs —

Cache operations: fetch 4.07 µs (ETS backend: 3.64 µs), evict_scope 0.15 µs.

Threshold: 64 KiB. The BEAM's guidance is ~1 ms per NIF call; on this machine that is crossed around 700 KiB, so 64 KiB carries a ~20x margin. It is not set higher because AES without hardware acceleration is roughly an order of magnitude slower — 64 KiB still lands under 1 ms on such a CPU, 256 KiB would not. Dirty dispatch costs a fixed ~0.5–1 µs, which is why the threshold is not simply zero. Every realistic encrypted column is far below 64 KiB, so in practice AshVault field operations never pay the dispatch cost.

Cache reads are a DashMap lookup — sub-microsecond of actual work — and stay on a normal scheduler unconditionally.

Override for unusual hardware:

config :ash_vault_rustler, dirty_threshold_bytes: 262_144

Note the honest result in that table: for binary keys, this cipher is not faster than OTP's, and at 1 MiB it is slower. Performance is not the reason to adopt it; the zeroing cache and the opaque key path are.


Safety

Known coverage gap

AshVault.KeyProviders.Cached.evict_scope/2 fans out to Node.list(:connected), which is empty on a single node, so the :erpc.multicall/5 call itself is never executed by the test suite — there is no cluster here to stand one up against. What is tested is the decision the fan-out feeds: AshVault.KeyProviders.Cached.classify_evictions/2 is public and unit-tested against every shape :erpc produces ({:ok, :ok}, {:ok, {:error, _}}, {:error, {:erpc, :noconnection}}, {:error, {:erpc, :timeout}}, {:exit, _}, {:throw, _}, an exception result, and garbage), asserting that anything other than a positive {:ok, :ok} is a failure and therefore makes destroy/1 return an error. The wiring between multicall and that function is four lines and is not covered.


Distribution

Consumers download a precompiled NIF through rustler_precompiled and verify it against checksum-Elixir.AshVaultRustler.Native.exs. A Rust toolchain is needed only when ASH_VAULT_RUSTLER_BUILD=1 forces a source build. The setting is read at compile time; run mix deps.compile ash_vault_rustler --force after changing it.

Artifacts cover Linux GNU and musl, plus macOS, on x86_64 and aarch64. Each uses NIF API 2.16, supported by OTP 24 and later. The package's Elixir requirement still applies.

Releasing both packages

  1. Update the versions in both Mix projects and the native crate, plus the changelog. Run the local suites and the release workflow with dry_run=true before merging.
  2. Push the matching v<version> tag, for example v0.2.0. The release workflow tests both packages, builds all six NIF artifacts and publishes ash_vault to Hex.
  3. The workflow generates checksums from the published artifacts, includes them in the ash_vault_rustler Hex package, and opens a PR to retain the checksum file in Git. Merge that PR after checking the artifact names and checksums.
  4. Verify both Hex releases and compile the published Rustler package from a clean consumer without forcing a source build. Verify that all six artifacts exist under ash_vault_rustler-v<version>.

Never rebuild or overwrite artifacts of an existing release: a different binary would invalidate the checksums already shipped to consumers. The release workflow refuses partial artifact releases rather than mixing builds.


Building and testing

Everything needs a linker. In this repo:

nix develop --command bash -c 'cd ash_vault_rustler && mix test'
nix develop --command bash -c 'cd ash_vault_rustler/native/ashvault_nif && cargo test'
nix develop --command bash -c 'cd ash_vault_rustler/native/ashvault_nif && cargo clippy --all-targets -- -D warnings'
nix develop --command bash -c 'cd ash_vault_rustler && mix run bench/scheduler_bench.exs'

The Elixir suite runs the parent's own AshVault.KeyProvider contract cases against AshVault.KeyProviders.Cached with both backends. test/parent_support/ holds a symlink to the parent's key_provider_cases.ex — the same file, not a copy, because a copied contract suite drifts and a drifted contract suite is worse than none.