slate

Type-safe Gleam wrapper for Erlang DETS (Disk Erlang Term Storage).

DETS stores key-value pairs in files. Stored data persists across node restarts. An abnormal node shutdown can lose pending writes and leave the file needing repair. DETS is built into OTP. No separate database service is required.

Important

Erlang target only. slate does not support the JavaScript target.

When to use DETS

Approach	Setup	Persistence	Query capability
JSON file	File I/O and serialization	Yes	None
DETS	Included in OTP; add slate for Gleam	Yes	Key lookup, fold
SQLite	Embedded database library	Yes	Full SQL
Postgres	Database server	Yes	Full SQL
Mnesia	Included in OTP; configure a schema	Yes	Key lookup, query expressions

Use DETS when you need persistent key lookup without SQL or distributed storage.

Installation

gleam add slate

Exclusive sessions use a native component included in Hex releases. They require OTP 26+ and a supported OS, architecture, and local filesystem. Source checkouts need Rust/Cargo 1.91 and just native-build. Unlocked APIs do not load native code. See native delivery and escripts.

Usage

An entry is one key-value pair. The examples use files in a data directory. Create it before you run them; slate does not create parent directories:

mkdir -p data

Set tables (one value per key)

import gleam/dynamic/decode
import slate/set

pub fn main() {
  // Open or create a table
  let assert Ok(users) = set.open("data/users.dets",
    key_decoder: decode.string, value_decoder: decode.int)

  // Insert key-value pairs
  let assert Ok(Nil) = set.insert(users, "alice", 42)
  let assert Ok(Nil) = set.insert(users, "bob", 37)

  // Look up values
  let assert Ok(age) = set.lookup(users, key: "alice")
  // age == 42

  // Check membership
  let assert Ok(True) = set.member(users, key: "alice")
  let assert Ok(False) = set.member(users, key: "charlie")

  // Always close when done
  let assert Ok(Nil) = set.close(users)
}

Open and close tables with `with_table`

import gleam/dynamic/decode
import slate
import slate/set

pub fn main() {
  // Attempt to close the table after the callback returns
  let assert Ok(Nil) = set.with_table("data/config.dets",
    repair: slate.AutoRepair, access: slate.ReadWrite,
    key_decoder: decode.string, value_decoder: decode.string,
    callback: fn(table) {
      set.insert(table, "theme", "dark")
    })
}

Use with_table for short-lived operations. Select the repair policy and access mode for each call. The callback must return a Result. If the callback succeeds but closing fails, the helper returns the close error. If both fail, it returns the callback error. If the callback raises, the helper attempts cleanup before re-raising the exception. A forced process termination can prevent helper cleanup.

DETS tracks the processes that open a table and normally closes it when the last user closes it or exits. An abnormal node shutdown can leave the file needing repair.

For read-only access without automatic repair:

import gleam/dynamic/decode
import slate
import slate/set

use table <- set.with_table(
  path: "data/config.dets", repair: slate.NoRepair, access: slate.ReadOnly,
  key_decoder: decode.string, value_decoder: decode.string)
set.lookup(table, key: "theme")

ReadOnly requires an existing file. This helper is available in all three table modules.

Migrating from 1.x: with_table now requires repair and access. Add repair: slate.AutoRepair and access: slate.ReadWrite to keep the previous behavior. Cleanup and error handling are unchanged.

Exclusive sessions across CLI processes

Use with_exclusive_table when independent BEAM VMs share a DETS file. The helper acquires an OS lock before opening or repairing DETS, then closes DETS before releasing ownership:

import gleam/dynamic/decode
import slate
import slate/exclusive
import slate/set

use table <- set.with_exclusive_table(
  path: "data/cache.dets",
  repair: slate.AutoRepair,
  access: slate.ReadWrite,
  wait: exclusive.WaitUpTo(1000),
  key_decoder: decode.string,
  value_decoder: decode.string,
)
set.insert(table, "theme", "dark")

Use Immediate for a single attempt. A contender receives AcquisitionFailed(Busy) or AcquisitionFailed(TimedOut) before DETS opens or the callback runs. The acquisition deadline does not cancel an acquired session.

For CLI caching, take a locked read session, fetch missing metadata without a table or lock, then take a new locked write session. Insert into the current table rather than replacing it with an earlier snapshot. Different-key writes accumulate; same-key set inserts keep last-write-wins behavior. Keep freshness checks and read-modify-write operations inside one callback.

All clients, including readers, must participate. Unlocked APIs, raw DETS, and older Slate versions ignore this advisory protocol. Keep table handles and all table work inside the callback. Do not reopen or close the table yourself. Callback errors do not undo writes. CleanupFailed preserves primary and cleanup failures; unsafe cleanup can retain ownership until the VM exits.

The persistent <resolved-path>.slate.lock sidecar records lock identity, not ownership. Do not delete or replace it while clients can start sessions. Symlinks resolve to the same lock; hard-linked data and unsafe sidecars return typed errors. Keep paths in a trusted, stable directory.

The helper is available for all three table types. See the exclusive-session guide for errors, recovery, filesystem limits, and measured open/close costs.

Bag tables (multiple values per key)

import gleam/dynamic/decode
import slate/bag

pub fn main() {
  let assert Ok(tags) = bag.open("data/tags.dets",
    key_decoder: decode.string, value_decoder: decode.string)

  let assert Ok(Nil) = bag.insert(tags, "color", "red")
  let assert Ok(Nil) = bag.insert(tags, "color", "blue")

  let assert Ok(colors) = bag.lookup(tags, key: "color")
  // Contains "red" and "blue" in an unspecified order

  let assert Ok(Nil) = bag.close(tags)
}

Duplicate bag tables

import gleam/dynamic/decode
import slate/duplicate_bag

pub fn main() {
  let assert Ok(events) = duplicate_bag.open("data/events.dets",
    key_decoder: decode.string, value_decoder: decode.string)

  let assert Ok(Nil) = duplicate_bag.insert(events, "click", "button_a")
  let assert Ok(Nil) = duplicate_bag.insert(events, "click", "button_a")

  let assert Ok(clicks) = duplicate_bag.lookup(events, key: "click")
  // Contains two "button_a" values; lookup does not guarantee an order

  let assert Ok(Nil) = duplicate_bag.close(events)
}

Data persists across restarts

import gleam/dynamic/decode
import slate/set

pub fn write() {
  let assert Ok(table) = set.open("data/state.dets",
    key_decoder: decode.string, value_decoder: decode.int)
  let assert Ok(Nil) = set.insert(table, "counter", 42)
  let assert Ok(Nil) = set.close(table)
}

pub fn read() {
  let assert Ok(table) = set.open("data/state.dets",
    key_decoder: decode.string, value_decoder: decode.int)
  let assert Ok(42) = set.lookup(table, key: "counter")
  let assert Ok(Nil) = set.close(table)
}

Error handling

Most public operations return Result(_, slate.DetsError).

slate/set.update_counter returns Result(_, set.UpdateCounterError). This type includes set.CounterValueNotInteger without adding a counter-specific variant to DetsError.

with_exclusive_table returns Result(_, exclusive.SessionError). Use exclusive.error_code and exclusive.error_message for these errors. The callback still returns Result(_, slate.DetsError).

The error_code and error_message helpers return strings. Exceptions raised by your callbacks propagate through with_table, fold, and fold_results.

Match on the specific variants you expect in normal flows, and use the helper functions when you want a stable code or a user-facing message:

import slate
import slate/set

case set.lookup(table, key: "missing") {
  Ok(value) -> Ok(value)
  Error(slate.NotFound) -> Ok(default_value)
  Error(error) -> {
    let code = slate.error_code(error)
    let message = slate.error_message(error)
    // log code/message here
    Error(error)
  }
}

UnexpectedError(detail) is intended for diagnostics only; the detail string is not a stable API contract, and error_message intentionally returns a generic message for that variant.

When opening existing files, Error(slate.NotADetsFile(context)) means the path is readable but not a DETS file, and Error(slate.NeedsRepair(context)) means the file was not closed cleanly and you opened it with NoRepair.

File-related errors include FileErrorContext(path: Option(String), reason: String). The path comes from OTP; None means OTP supplied no single filename. The reason retains lower-level diagnostics, including nested OTP reasons. Both fields are for trusted diagnostics only and can contain sensitive data. error_code and error_message do not include these details.

Breaking migration: FileNotFound, AlreadyOpen, AccessDenied, TypeMismatch, NeedsRepair, NotADetsFile, and FileSizeLimitExceeded now require context. Change patterns such as Error(slate.AccessDenied) to Error(slate.AccessDenied(_)), or bind context to inspect it. See the migration guide for before/after examples, including code that constructs errors. No DETS file migration is needed.

For set.update_counter, match Error(set.CounterValueNotInteger) directly and unwrap shared table failures as Error(set.TableError(error)).

API overview

The three table types (set, bag, duplicate_bag) share a common core API:

Function	Description
`open(path, key_decoder, value_decoder)`	Open or create a table
`open_with(path, repair, key_decoder, value_decoder)`	Open with repair policy
`open_with_access(path, repair, access, key_decoder, value_decoder)`	Open with repair and access mode
`close(table)`	Close and flush to disk
`sync(table)`	Flush without closing
`with_table(path, repair, access, key_decoder, value_decoder, fn)`	Auto-closing callback with repair and access options
`with_exclusive_table(path, repair, access, wait, key_decoder, value_decoder, fn)`	Cross-process exclusive callback, closed before unlock
`insert(table, key, value)`	Insert a key-value pair
`insert_list(table, entries)`	Batch insert
`lookup(table, key)`	Get value(s) for key
`member(table, key)`	Check if key exists
`delete_key(table, key)`	Remove by key
`delete_object(table, key, value)`	Remove a specific key-value pair (`duplicate_bag` removes all exact duplicates)
`delete_all(table)`	Clear all entries
`to_list(table)`	Get all entries
`fold(table, acc, fn)`	Fold over entries
`fold_results(table, acc, fn)`	Fold over entries and handle decode errors in the callback
`size(table)`	Count entries
`info(table)`	Get the file size in bytes, entry count, and absolute file path

Use info(table) to retrieve the file path from an open handle:

let assert Ok(info) = set.info(table)
info.file_path

Migrating from 1.x: TableInfo now has a file_path: String field. Supply it when constructing the record. Include the third field, or use .., when matching the constructor. Existing info.file_size and info.object_count access remains valid.

slate/set also provides:

Function	Description
`insert_new(table, key, value)`	Insert if key is absent
`update_counter(table, key, amount)`	Atomic counter increment

slate/bag also provides:

Function	Description
`insert_new(table, key, value)`	Return an error if the exact key-value pair exists

Concurrent bag.insert_new calls can both return Ok(Nil) for the same pair. The table still stores one copy. To ensure only one caller reports a new insertion, serialize writes through an owner process.

Bag and duplicate bag lookups return values in an unspecified order. A duplicate bag does not enforce append-only access. Store a timestamp or sequence number if you need to reconstruct event order.

size counts entries, not distinct keys. Each duplicate copy counts as an entry. info.file_size is the file size in bytes; info.object_count is the entry count.

The top-level slate module also provides:

Function	Description
`is_dets_file(path)`	Check if a file is a valid DETS file
`error_code(error)`	Stable machine-readable error code
`error_message(error)`	User-facing error message

Limitations

Each table has a 2 GB file size limit.
DETS supports set, bag, and duplicate_bag, but not ordered_set.
Disk-backed access is slower than ETS for frequent reads.
Close tables after use. with_table attempts cleanup when the callback returns or raises.
The internal table name pool is bounded. Opening too many distinct tables at once can fail with TableNamePoolExhausted. Close unused tables to release their slots.
slate supports only Gleam's Erlang target.
Independent VMs must use exclusive sessions for every access to a shared file. Simultaneous unlocked access can lose writes or damage the file. Exclusive sessions require qualified local storage; network filesystems are unsupported.

By default, DETS saves a table after three minutes without table access. Call sync and handle its result when you need to write pending updates to disk before continuing.

Make a backup before you repair a file with data you need to keep. Repair does not guarantee recovery of all data. Handle NeedsRepair separately from missing files, permission failures, and other open errors.

Stability

slate follows Semantic Versioning. The public API covered by semver guarantees consists of five modules:

slate: Shared types (DetsError, FileErrorContext, AccessMode, RepairPolicy, TableInfo) and helpers
slate/set: Set tables
slate/bag: Bag tables
slate/duplicate_bag: Duplicate bag tables
slate/exclusive: Acquisition waits, session errors, and safe classifiers

The Erlang FFI files and native lock implementation are internal details and are not part of the public API. They may change in any release without notice.

Versioning policy: patch releases contain bug fixes only, minor releases add backward-compatible features, and major releases may include breaking changes. The error_code() strings returned by slate.error_code are stable across minor and patch releases and are safe for programmatic matching (e.g., in error-handling logic or logging). The error_message() strings are human-readable and may change in any release.

See CHANGELOG.md for release history and upgrade notes, and the GitHub Releases page for tagged versions.

bravo provides ETS bindings for Gleam.
shelf provides persistent ETS tables backed by DETS. It uses slate for disk storage and ETS for in-memory reads.

For details on the underlying storage engine, see the Erlang DETS documentation.

Development

See DEV.md for setup instructions, build tasks, and contribution guidelines.

License

MIT