ash_vault_rustler
A Rust NIF providing an AshVault key cache and an AES-256-GCM cipher that hold key material outside the BEAM heap, so that eviction can actually zero it.
Optional. The parent ash_vault has no Rust dependency and never will.
def deps do
[
{:ash_vault, "~> 0.2.0"},
{:ash_vault_rustler, "~> 0.2.0"}
]
end
use AshVault.Vault,
key_provider: MyApp.Keys,
cache: [backend: AshVaultRustler.KeyCache, ttl: :timer.seconds(30)]
What this buys
The BEAM cannot zero memory, and that makes crypto-erasure eventual
AshVault's whole reason to exist is one sentence: destroying a customer's keys makes
their historical ciphertext permanently undecryptable. A key cache quietly weakens that.
"Evicting" a key from ETS drops a reference; the bytes survive in however many process
heaps read them, for however long the garbage collector takes, and there is no BEAM API
that overwrites them. Every BEAM crypto library shares this. Cloak does not attempt it —
zero hits for zero/wipe/scrub across its lib/, and it keeps every key in a named
:protected ETS table for the life of the process.
AshVaultRustler.KeyCache fixes exactly that, and only that:
AshVault.KeyCaches.ETS |
AshVaultRustler.KeyCache |
|
|---|---|---|
| Eviction is synchronous | yes | yes |
| Eviction is observable (no later read can see it) | yes | yes |
| Generation fence against a racing repopulate | yes | yes |
| Eviction overwrites the bytes | no | yes |
Key pages mlocked out of swap |
no | yes (best effort) |
| Needs a Rust toolchain or a precompiled artifact | no | yes |
Each cached key lives in a page-aligned allocation the crate owns, mlocked into RAM,
and Dropped with a volatile overwrite. evict_scope/1 returns only after those writes
have happened.
Opaque key handles
AshVaultRustler.KeyProviders.Opaque wraps any provider so it returns
%AshVault.Key{} handles instead of binaries, and AshVaultRustler.Cipher runs the AEAD
against a handle. The key never becomes an Elixir term during an encrypt or a decrypt.
A cipher you can drop in without stranding a row
AshVaultRustler.Cipher uses the same id (:aes_256_gcm_v1), the same 12-byte nonce,
the same 16-byte tag and the same AAD as AshVault.Ciphers.AES.GCM. A value written by
either decrypts with the other; the test suite asserts both directions across sizes from
0 bytes to 200 KB.
What this does not buy
Be precise about this, because the temptation to overclaim here is enormous.
The level-1 cache does not eliminate the per-operation copy.
AshVault.KeyProviders.Cached.get_key/3 returns a binary to the BEAM. A copy of the
key lands on an Elixir process heap for every single encrypt and decrypt, and that copy
is the garbage collector's business, not ours. This package bounds the lifetime of the
authoritative copy. Eliminating the per-operation one needs the opaque-handle path,
which is a separate opt-in.
Even the opaque path has a boundary. The wrapped provider returns a binary — that is its contract — so the key bytes pass through the BEAM once, on the way into the handle. What goes away is the repeat: with a cache in front, key bytes exist as an Elixir term once per cache miss rather than once per field read.
The TTL is the erasure SLA, but only for erasures that do not go through the vault.
AshVault.KeyProviders.Cached.destroy/2 evicts synchronously on every connected node
before it returns, and fails if any node does not acknowledge — so an erasure performed
through the vault is immediate. An erasure performed out of band is not: a
mix ash_vault.destroy_keys in an unclustered node, an operator deleting a transit key
in OpenBao by hand, another service revoking it. None of those call evict_scope/1, and
the cache serves the erased scope until the entry expires. That TTL is the number you
can write in a DPA, and it defaults to 30 seconds.
mlock is best effort, and it is charged per page. Each cached key gets its own
page, because mlock/munlock act on whole pages and locking a 32-byte buffer inside a
shared page would unlock whatever else lives there when it is freed. So:
A 32-byte key costs 4 KiB of locked memory, and
:max_bytescounts key bytes, not pages. The defaults (max_entries: 1_024,max_bytes: 1_048_576) can lock up to 4 MiB while the reported byte total reads 32 KiB. With this backend, size:max_entriesagainstRLIMIT_MEMLOCK, not against:max_bytes.
A default container ulimit -l is often 64 KiB — sixteen pages — so locking starts
failing after roughly sixteen cached keys on a stock host. That is not fatal: the failure
is logged once, loudly, after the first write that observes it, and caching continues,
because a key that might reach swap beats a node that refuses to decrypt. Check
AshVaultRustler.mlock_status/0 in production and alert on a non-zero :failed.
Nothing here defends against a compromised application. An attacker with code execution can ask the cache for a key exactly as your app does. See the parent's threat model.
The rule that matters most
Never cache the absence of a tombstone.
{:error, :destroyed} is cached forever: destruction is monotonic, so a cached tombstone
can only be right, and caching it makes reads fail closed even when the provider is
unreachable. The absence of a tombstone is never cached; nor is {:error, :not_found},
nor a transport failure.
Caching {:ok, key} is, strictly, an implicit cache of "not destroyed", and pretending
otherwise would be the fifth fail-open tombstone read in this project's history. Two
mechanisms contain it: synchronous cluster-wide eviction on the destroy path, and a short
TTL as the floor for everything else. The policy lives in
AshVault.KeyProviders.Cached — in Elixir, where it can be read and reviewed — and the
Rust crate holds no opinion about tombstones at all.
Configuration
The cipher: set both :cipher and :ciphers
This is the easiest thing to get wrong, and it is a consequence of the wire compatibility being real:
# governs ENCRYPTION
use AshVault.Vault, key_provider: MyApp.Keys, cipher: AshVaultRustler.Cipher
# governs DECRYPTION — required
config :ash_vault, :ciphers, %{"aes_256_gcm_v1" => AshVaultRustler.Cipher}
AshVault.Vault.Runtime.decrypt!/3 resolves the cipher from the envelope, through
AshVault.Cipher.registry/0, not from the vault's :cipher option — deliberately, so
that values keep decrypting after the default cipher changes. Since this cipher shares
the built-in id, a vault configured with only :cipher will encrypt in Rust and decrypt
in Elixir. Harmless with a binary key; impossible with an opaque handle, which is
where you find out.
The cache
use AshVault.Vault,
key_provider: MyApp.Keys,
cache: [
backend: AshVaultRustler.KeyCache,
ttl: :timer.seconds(30), # current key; also the erasure SLA
historical_ttl: :timer.seconds(30),
max_entries: 1_024,
max_bytes: 1_048_576,
cluster: true, # evict on every connected node, fail if any doesn't ack
evict_timeout: 5_000
]
Then supervise the generated provider — it owns the cache:
children = [MyApp.Vault.CachedKeyProvider]
If it is not started, every read goes to the provider. Slower, and correct.
Opaque keys
defmodule MyApp.OpaqueKeys do
use AshVaultRustler.KeyProviders.Opaque, provider: AshVault.KeyProviders.OpenBao
end
use AshVault.Vault, key_provider: MyApp.OpaqueKeys, cipher: AshVaultRustler.Cipher
config :ash_vault, :ciphers, %{"aes_256_gcm_v1" => AshVaultRustler.Cipher}
A cipher that cannot take a handle raises AshVault.Errors.OpaqueKeyUnsupported naming
both modules. There is deliberately no fallback that unwraps the handle — that would put
the key back on the BEAM heap, silently, which is the one thing the handle prevents.
Scheduler discipline, and the numbers behind it
Measured on this host (AES-NI, mix run bench/scheduler_bench.exs):
| plaintext | normal scheduler | dirty CPU scheduler | OTP :crypto |
|---|---|---|---|
| 16 B | 1.20 µs | 1.73 µs | 1.90 µs |
| 64 B | 0.68 µs | — | — |
| 1 KiB | 1.70 µs | 3.04 µs | 2.72 µs |
| 4 KiB | 3.92 µs | — | — |
| 16 KiB | 13.6 µs | — | — |
| 64 KiB | 46.4 µs | 80.9 µs | 42.0 µs |
| 256 KiB | 319 µs | — | — |
| 1 MiB | 1529 µs | 1055 µs | 715 µs |
| 4 MiB | 5958 µs | 5392 µs | — |
Cache operations: fetch 4.07 µs (ETS backend: 3.64 µs), evict_scope 0.15 µs.
Threshold: 64 KiB. The BEAM's guidance is ~1 ms per NIF call; on this machine that is crossed around 700 KiB, so 64 KiB carries a ~20x margin. It is not set higher because AES without hardware acceleration is roughly an order of magnitude slower — 64 KiB still lands under 1 ms on such a CPU, 256 KiB would not. Dirty dispatch costs a fixed ~0.5–1 µs, which is why the threshold is not simply zero. Every realistic encrypted column is far below 64 KiB, so in practice AshVault field operations never pay the dispatch cost.
Cache reads are a DashMap lookup — sub-microsecond of actual work — and stay on a
normal scheduler unconditionally.
Override for unusual hardware:
config :ash_vault_rustler, dirty_threshold_bytes: 262_144
Note the honest result in that table: for binary keys, this cipher is not faster than OTP's, and at 1 MiB it is slower. Performance is not the reason to adopt it; the zeroing cache and the opaque key path are.
Safety
- No hand-rolled crypto. RustCrypto's
aes-gcm, which has had a third-party audit (NCC Group, funded by MobileCoin) and uses AES-NI/CLMUL where available. - No NIF can abort the VM. Every NIF returns a value or an error term. Rustler wraps
each body in
catch_unwind, so a panic becomes an error in the calling process;test/panic_test.exsproves that on this OTP build rather than asserting it. unsafeappears in exactly one module (native/ashvault_nif/src/secret.rs), for the page-aligned allocation and its zeroing drop. Every block carries aSAFETY:comment, because a bug there takes down the node rather than one process.- Constant-time comparison (
subtle) for anything whose result is a fact about key material. - Truncated tags and short nonces are refused before the key reaches the AEAD — the
same control, for the same stated reason, as the Elixir cipher. OTP's
:crypto.crypto_one_time_aead/7accepts a truncated GCM tag and compares only its leading bytes, which made a ≤256-guess forgery possible through an attacker-controlledtag_lenin the envelope. - Unauthenticated plaintext is zeroed rather than returned when a tag check fails.
Known coverage gap
AshVault.KeyProviders.Cached.evict_scope/2 fans out to Node.list(:connected), which is
empty on a single node, so the :erpc.multicall/5 call itself is never executed by the
test suite — there is no cluster here to stand one up against. What is tested is the
decision the fan-out feeds:
AshVault.KeyProviders.Cached.classify_evictions/2 is public and unit-tested against
every shape :erpc produces ({:ok, :ok}, {:ok, {:error, _}}, {:error, {:erpc, :noconnection}}, {:error, {:erpc, :timeout}}, {:exit, _}, {:throw, _}, an
exception result, and garbage), asserting that anything other than a positive
{:ok, :ok} is a failure and therefore makes destroy/1 return an error. The wiring
between multicall and that function is four lines and is not covered.
Distribution
Consumers download a precompiled NIF through rustler_precompiled and verify it against
checksum-Elixir.AshVaultRustler.Native.exs. A Rust toolchain is needed only when
ASH_VAULT_RUSTLER_BUILD=1 forces a source build. The setting is read at compile time;
run mix deps.compile ash_vault_rustler --force after changing it.
Artifacts cover Linux GNU and musl, plus macOS, on x86_64 and aarch64. Each uses NIF API 2.16, supported by OTP 24 and later. The package's Elixir requirement still applies.
Releasing both packages
- Update the versions in both Mix projects and the native crate, plus the changelog.
Run the local suites and the release workflow with
dry_run=truebefore merging. - Push the matching
v<version>tag, for examplev0.2.0. The release workflow tests both packages, builds all six NIF artifacts and publishesash_vaultto Hex. - The workflow generates checksums from the published artifacts, includes them in the
ash_vault_rustlerHex package, and opens a PR to retain the checksum file in Git. Merge that PR after checking the artifact names and checksums. - Verify both Hex releases and compile the published Rustler package from a clean
consumer without forcing a source build. Verify that all six artifacts exist under
ash_vault_rustler-v<version>.
Never rebuild or overwrite artifacts of an existing release: a different binary would invalidate the checksums already shipped to consumers. The release workflow refuses partial artifact releases rather than mixing builds.
Building and testing
Everything needs a linker. In this repo:
nix develop --command bash -c 'cd ash_vault_rustler && mix test'
nix develop --command bash -c 'cd ash_vault_rustler/native/ashvault_nif && cargo test'
nix develop --command bash -c 'cd ash_vault_rustler/native/ashvault_nif && cargo clippy --all-targets -- -D warnings'
nix develop --command bash -c 'cd ash_vault_rustler && mix run bench/scheduler_bench.exs'
The Elixir suite runs the parent's own AshVault.KeyProvider contract cases against
AshVault.KeyProviders.Cached with both backends. test/parent_support/ holds a symlink
to the parent's key_provider_cases.ex — the same file, not a copy, because a copied
contract suite drifts and a drifted contract suite is worse than none.