slap_snapshot_log

slap_snapshot_log provides an append-only log of changes that can be condensed into snapshots. A consumer rebuilds its state from the latest snapshot and all subsequent entries, and follows the log to stay current. The log is stored in Durable Streams, using slap_streams, under a base path of your choosing.

Use slap_snapshot_log when an application's state comes from an ordered log of changes, but replaying every change from the beginning is too slow. For example, a counter can append one increment per entry and periodically store its total as a snapshot. A new reader loads that total, then applies only later entries. The package stores and trims the log; your application defines the entry format, how entries change state, and how to encode a snapshot. If you need to keep every entry, or don't need snapshots, use slap_streams directly.

About Slap {: .info}

This package is part of Slap.

Slap provides Elixir bindings for SlateDB (an embedded key-value engine optimized for storing data in object storage), as well as services built on top of these bindings. The services run inside your application, on one node or several.

Services:

Building blocks:

Server distribution:

Inspection:

The slap package pulls in all seven server libraries. If you only need a subset, add those packages directly.

Installation

Requires Elixir 1.18 or later.

To depend on slap_snapshot_log directly, add it to the dependencies in your application's mix.exs:

defp deps do
  [
    {:slap_snapshot_log, "~> 0.5.0"}
  ]
end

Run mix deps.get. API documentation is on HexDocs.

Example

In an application that depends on slap_snapshot_log, start Slap.Streams.Cluster before using the log. For example, in iex -S mix with a fresh local directory:

iex> {:ok, _} = Slap.Streams.Cluster.start_link(store: {:local, "/tmp/slap-snapshot-log"}, shards: 8); :ok
:ok
iex> base = "/v1/stream/docs/42"
"/v1/stream/docs/42"

iex> {:reset, %{snapshot: nil, offset: 0}} = Slap.SnapshotLog.next(base, nil); :reset
:reset
iex> {:ok, _} = Slap.SnapshotLog.append(base, "entry"); :ok
:ok
iex> {:ok, %{entries: entries, offset: offset}} = Slap.SnapshotLog.next(base, 0); entries
["entry"]
iex> state_bytes = "entry"
"entry"
iex> :ok = Slap.SnapshotLog.snapshot(base, offset, state_bytes)
:ok

The application defines how to encode its state in state_bytes. See slap_streams for cluster configuration. An application-owned Streams cluster can be passed as cluster: to log operations. Run one Streams cluster per VM.

Reading

next(base, offset, opts) returns one of:

A consumer handles a reset the same way at start and later: another process's snapshot can trim entries the consumer has not read yet.

Appending

append(base, entry) returns {:ok, offset}, the offset after the entry, once the entry is durable. It creates the log if needed. It does not retry: on {:error, :unavailable} or {:error, :timeout} the entry may still be stored. To retry without appending the entry twice, pass the same producer: {id, epoch, seq} to each attempt. A retry returns {:duplicate, current_tail} if that sequence was already written.

Snapshots

snapshot(base, offset, bytes, opts) publishes a snapshot that replaces the entries before offset, makes it current, and trims those entries. Snapshot publication, listing, reading, tail lookup, and deletion accept timeout: to bound each Streams call (default 30 s).

A snapshot at offset must contain the effect of every entry before offset. It may contain more only if applying those entries again, when the entries from offset on are replayed, is harmless. A consumer whose entries cannot be applied twice snapshots only the entries it has read from the log. A consumer whose entries are idempotent, such as Yjs updates, may snapshot its live state. The log cannot check this.

Several processes can publish at once. Only the snapshot at the highest offset becomes current; the others return {:error, :superseded}. Each step of a publication can be interrupted: the log stays readable, and a later publication, or a retry at the same offset, deletes what the interrupted one left. A snapshot at an offset after the tail is {:error, :offset_beyond_tail}.

The :history option keeps older snapshots: a list of {every_ms, keep_ms} rules, each keeping the latest snapshot of every every_ms period for keep_ms. For example, history: [{3_600_000, 7 * 24 * 3_600_000}] keeps the latest snapshot from each hour for up to a week. snapshots/1 lists the stored snapshots and read_snapshot/2 reads one.

Forking

fork(source_base, target_base, opts) creates a log at target_base that starts with the source's contents: its current snapshot, if any, and the entries after it. From then on the two logs are independent; entries appended to one do not appear in the other. The target uses its own base as its placement key, so it can be on another shard.

{:ok, %{offset: offset, snapshot_offset: snapshot_offset}} =
  Slap.SnapshotLog.fork("/logs/main", "/logs/branch", timeout: 30_000)
{:ok, _} = Slap.SnapshotLog.append("/logs/branch", "branch entry")

offset is the source's tail when the fork was made, and the target has the source's entries before it. The target's current snapshot is the source's snapshot at snapshot_offset (nil if the source had none). There is no option to fork from an earlier offset.

fork/3 reads metadata only. The target refers to the source's snapshot and entries instead of copying them, and they stay stored while the target needs them, even after the source is deleted. Reading the target therefore needs the shards of the source, and of the source's own sources if it is a fork. Publishing a snapshot on the target trims the target's entries but does not release the source entries it refers to; they stay stored until the target is deleted.

Options:

{:error, :timeout}, :unavailable, :overloaded and :busy (the source is still being created or forked) leave the fork unfinished. Call fork/3 again with the same source and target to finish it. Until then, other calls on the target return {:error, :busy}. Once the fork has finished, calling fork/3 again with the same source and target returns the same result, even if the source has since been deleted or the target edited. delete/1 on an unfinished target cancels the fork; later fork/3 calls with that target return the error that cancelled it, so start again with a new target base.

Other errors:

A snapshot can contain more than the entries before its offset (see "Snapshots"), and the target starts with whatever the source's current snapshot contains. fork/3 does not contact the processes that append to the source; anything they have not appended yet is not in the target. slap_yjs explains how to include a Yjs document server's buffered updates.

Deleting

delete(base) deletes the log permanently. Its entries and snapshots are deleted, and every later call on the base returns {:error, :deleted}. That includes a next/3 waiting at the tail, an append/2, and a snapshot publication that started before the delete.

The base cannot be reused. A deleted log keeps these records:

A log created by fork/3 has its other streams deleted. Data it shares with its source, or with forks made from it, stays stored while any of those logs needs it.

The log reads as deleted from the delete's first step. Retrying delete/1 finishes an interrupted delete.

Storage

<base>/.updates                      the entries, one message each
<base>/.index                        JSON snapshot metadata or a deletion marker
<base>/.snapshots/<offset>_snapshot  a snapshot
<base>/.control                      whether the log is being created, forked or deleted

Snapshot metadata contains "snapshotOffset", "createdAt", and "retained"; before deletion, the last snapshot entry is current. The first three streams are the reference y-durable-streams server's layout ("retained" is an addition). Logs created by earlier versions of slap_snapshot_log may have no .control stream; they still work. The streams share base as their placement key, so they are on one shard. base must be a stream path with no segment that starts with ..

While a log is being created or forked, calls on it return {:error, :busy}, except that append/3 and next/3 finish a creation that another caller left unfinished.

Tests

Follow the repository's development setup, then run from slap_snapshot_log/:

mix test

The tests cover reading, appending, snapshot publication (including interrupted and competing publications), snapshot history, and deletion.

License

Slap is released under the terms of the Apache License 2.0.

Copyright (c) 2026, Michael Russo.