DurableServer

DurableServer provides durable, distributed GenServer and gen_statem processes backed by pluggable storage backends.

It implements fault-tolerant, stateful processes that can survive node failures, restarts, and deployments by automatically persisting state and coordinating across a distributed cluster.

Key Features

Installation

Add durable_server to your list of dependencies in mix.exs:

def deps do
  [
    {:durable_server, "~> 0.1.8"}
  ]
end

For releases, add :os_mon to extra_applications:

def application do
  [
    mod: {MyApp.Application, []},
    extra_applications: [:logger, :runtime_tools, :os_mon]
  ]
end

Basic Usage

defmodule MyCounterServer do
  use DurableServer, vsn: 1

  def dump_state(state), do: %{count: state.count}

  def load_state(_old_vsn, %{"count" => count}), do: %{count: count}

  def init(%{count: count} = state) do
    {:ok, Map.merge(state, %{started_at: DateTime.utc_now()})}
  end

  def handle_call(:increment, _from, state) do
    new_state = %{state | count: state.count + 1}
    {:reply, new_state.count, new_state}
  end

  def handle_call(:get_count, _from, state) do
    {:reply, state.count, state}
  end
end

Start the supervisor (typically in your application supervision tree):

children = [
  {DurableServer.Supervisor,
   name: MyDurableSup,
   prefix: "my_app/",
   object_store: [
     bucket: "my-bucket",
     access_key_id: System.fetch_env!("DURABLE_AWS_ACCESS_KEY_ID"),
     secret_access_key: System.fetch_env!("DURABLE_AWS_SECRET_ACCESS_KEY"),
     s3_endpoint: System.fetch_env!("DURABLE_AWS_ENDPOINT_URL_S3"),
     default_region: System.fetch_env!("DURABLE_AWS_REGION")
   ]}
]

Start and use individual servers:

{:ok, {pid, _meta}} = DurableServer.Supervisor.start_child(
  MyDurableSup,
  {MyCounterServer, key: "user_123", initial_state: %{count: 0}}
)

GenServer.call(pid, :increment)  # => 1
GenServer.call(pid, :increment)  # => 2
GenServer.call(pid, :get_count)  # => 2

For GenServers, :initial_state is required and must be a map. On first boot, DurableServer passes it through dump_state/1, the configured backend's encode/decode path, and then load_state/2 before init/1 or init/2. The dumped initial state must therefore be encodable by your configured backend.

State Machines

Opt in with use DurableServer, vsn: 1, behaviour: :gen_statem. The process runs on OTP's native gen_statem engine, with state entry callbacks, postponed events, inserted events, named timeouts, and state timeouts. Implement callback_mode/0 to select :handle_event_function or :state_functions, optionally with :state_enter, just as with native gen_statem. The :behaviour option accepts GenServer (the default) or :gen_statem.

defmodule DurableSwitch do
  use DurableServer, vsn: 1, behaviour: :gen_statem

  def callback_mode, do: :handle_event_function

  # The state name and data are persisted together, using your chosen encoding.
  def dump_state({state, count}), do: %{"state" => Atom.to_string(state), "count" => count}
  def load_state(_, %{"state" => "off", "count" => count}), do: {:off, count}
  def load_state(_, %{"state" => "on", "count" => count}), do: {:on, count}

  def init({state, count}) do
    {:ok, state, count, [{:durable, [permanent: true]}]}
  end

  def handle_event({:call, from}, :toggle, state, count) do
    next = if state == :off, do: :on, else: :off

    {:next_state, next, count + 1,
     [{:durable, [sync: true, meta: %{state: next}]}, {:reply, from, next}]}
  end
end

{:ok, {pid, _meta}} = DurableServer.Supervisor.start_child(
  MyDurableSup,
  {DurableSwitch, key: "switch_123", initial_state: {:off, 0}}
)

:gen_statem.call(pid, :toggle) # => :on

:initial_state, load_state/2, and dump_state/1 use {state_name, data}. The data can be any term your serialization callbacks can encode into a plain map. Without a custom init/1, the loaded pair becomes the initial state and data. init/2 is also supported and receives the usual supervisor info map.

Use {:durable, options} in a native action list for durability options:

sync: true persists the transition before any native actions execute, including replies and :next_event. A failed explicit write terminates the owner without running those actions. handle_sync/3 receives the old and new state/data pairs and has the same post-write semantics as with GenServer. Native timers and postponed events remain in memory; reconstruct any required work on cold starts.

Use the supervisor's existing lifecycle API for stopping, deleting, and cordoning. Supervisor stop calls wait for terminate/3 and final persistence. A user callback's native :stop_and_reply retains OTP's reply-before-termination semantics. code_change/4 receives the state name and data separately for hot upgrades. For inspection, :sys.get_state/1 returns {state_name, %DurableServer{user_state: {state_name, data}}}.

Administrative Cordon

Use terminate_and_cordon_child/3 when you need to stop a DurableServer and make it ineligible for any future starts until an operator explicitly clears the cordon.

# Stop a running child and persist status: :cordoned
:ok = DurableServer.Supervisor.terminate_and_cordon_child(MyDurableSup, pid)

# Or cordon by key. If the child is not running, storage is updated directly.
:ok = DurableServer.Supervisor.terminate_and_cordon_child(
  MyDurableSup,
  "user_123",
  timeout: 10_000
)

# Later, allow explicit starts and LifecycleManager recovery again.
:ok = DurableServer.Supervisor.uncordon_child(MyDurableSup, "user_123")

A cordoned object stores status: :cordoned. While cordoned:

When cordoning by key, DurableServer uses the same lock/orphan checks as normal startup before updating storage. A live owner is asked to persist the cordoned status itself; an orphaned owner is only replaced after the lock is expired.

Storage Backends

DurableServer includes two built-in backends:

Object Storage Backend

{DurableServer.Supervisor,
 name: MyDurableSup,
 prefix: "my_app/",
 backend: {DurableServer.Backends.ObjectStore,
  [
    bucket: "my-bucket",
    access_key_id: "...",
    secret_access_key: "...",
    s3_endpoint: "...",
    default_region: "..."
  ]}}

EKV Backend

Start EKV in your application tree (CAS config is required for DurableServer lock semantics):

ekv_config = [
  name: :durable_ekv,
  data_dir: "/path/to/ekv_store",
  cluster_size: 3
]

children = [
  {EKV,
   name: :durable_ekv,
   data_dir: "/data/ekv/durable",
   cluster_size: 3,
   node_id: System.fetch_env!("EKV_NODE_ID")},
  {DurableServer.Supervisor,
   name: MyDurableSup,
   prefix: "my_app/",
   backend: {DurableServer.Backends.EKVStore, ekv_config}}
]

If you use EKV backend, add EKV to your app's dependencies.

Mirror Backend (Object Storage -> EKV)

Use the mirror backend to dual-write while you cut over reads/writes in phases.

See DurableServer.Backends.MirrorStore for usage and an example rollout.

Configuration Options

DurableServer supports these options in the init/1 return tuple:

State Synchronization

State is synchronized to storage in these scenarios:

  1. Manual sync: Return :sync from any callback: {:noreply, state, :sync}. Manual sync, {:sync, metadata}, and the sync: true callback option are strict durability boundaries. The backend retries classified transient failures within its bounded retry policy. If the write still fails, the server exits with a structured {:sync_failed, reason} fatal-exit reason before acknowledging the callback.
  2. Automatic sync: When :auto_sync is enabled, changes sync on callback return. Periodic sync uses the configured :sync_every_ms interval. After the backend exhausts transient retries, automatic and periodic sync log the failure and keep the dirty in-memory state eligible for a later sync. Storage conflicts remain fatal.
  3. Graceful shutdown: State is always synced before termination

Periodic and automatic sync skip dumping, serialization, and hashing when callback state is exactly unchanged and no earlier change is pending. Any runtime state change schedules a check of dump_state/1; if its output is unchanged, the check skips the storage write and clears the pending flag. Failed writes remain pending for a later sync.

dump_state/1 should deterministically derive its output from the supplied state. Changes to external data, such as ETS or the process dictionary, cannot be detected by state tracking; use an explicit sync to re-evaluate the dump when needed. Explicit sync and lifecycle writes always evaluate dump_state/1.

Manual synchronization performs storage work inline. A GenServer.call/3 timeout should be long enough to cover the configured backend's retry window. A caller timeout does not cancel storage work already running in the DurableServer process.

Group

Group provides distributed process groups, registry, lifecycle monitoring, and isolated subclusters.

Monitoring Events

Monitor lifecycle events for DurableServers:

# Monitor a specific key
:ok = Group.monitor(MyDurableSup, "user/123")

# Monitor all keys with a prefix
:ok = Group.monitor(MyDurableSup, "user/")

# Monitor all events
:ok = Group.monitor(MyDurableSup, :all)

Monitors receive {:group, events, info} tuples in their mailbox:

def handle_info({:group, events, _info}, state) do
  Enum.each(events, fn
    %Group.Event{type: :registered, key: key, pid: pid, previous_meta: nil} ->
      # A DurableServer started (previous_meta is nil for first registration)
      :ok
    %Group.Event{type: :unregistered, key: key, reason: reason} ->
      # A DurableServer stopped
      :ok
    _ -> :ok
  end)
  {:noreply, state}
end

Event types: :registered, :unregistered, :joined, :left

:registered and :joined events include a previous_meta field (nil for new, old meta for re-register/re-join). Single operations produce one event per tuple; bulk operations (nodedown, process death) batch all events together.

Joining as a Member

Non-DurableServer processes can join keys to be discoverable and receive dispatched messages:

# Join a key (e.g., from a Phoenix Channel)
:ok = Group.join(MyDurableSup, "room/123", %{type: :channel})

# Re-joining updates metadata in place
:ok = Group.join(MyDurableSup, "room/123", %{type: :channel, status: :active})

# Query all members of a key (DurableServers + joined processes)
members = Group.members(MyDurableSup, "room/123")
# => [{#PID<0.150.0>, %{...}}, {#PID<0.200.0>, %{type: :channel, status: :active}}]

# Leave when done (also happens automatically on process death)
:ok = Group.leave(MyDurableSup, "room/123")

Dispatching to Members

Send messages to all members of a key:

# From a DurableServer, broadcast to all connected channels
Group.dispatch(MyDurableSup, state.key, {:new_message, message})

Named Clusters

For advanced use cases, you can create isolated subclusters where only connected nodes receive events:

# Connect this node to a named cluster
:ok = Group.connect(MyDurableSup, :game_servers)

# Join/monitor/dispatch with the cluster: option
:ok = Group.join(MyDurableSup, "room/123", %{}, cluster: :game_servers)
:ok = Group.monitor(MyDurableSup, :all, cluster: :game_servers)

Note: DurableServers always register in the default cluster to ensure global uniqueness. Named clusters are purely for the pub/sub layer.

Monitor vs Join

These are independent - joining does not monitor events, and monitoring does not make you discoverable.

Running Tests

Unit Tests (with LocalStack)

Start LocalStack for S3-compatible storage:

docker run -d --name localstack -p 4566:4566 localstack/localstack

Run the tests:

mix test

Integration Tests (with Tigris)

Set the required environment variables:

Note: You can add these to a gitignored .env in this project and they will be loaded automatically in test_helper.exs

export DURABLE_AWS_ACCESS_KEY_ID=<your-tigris-access-key>
export DURABLE_AWS_SECRET_ACCESS_KEY=<your-tigris-secret-key>
export DURABLE_AWS_ENDPOINT_URL_S3=https://t3.storage.dev
export DURABLE_AWS_ENDPOINT_URL_IAM=https://iam.storage.dev
export DURABLE_AWS_REGION=<your-region>
export DURABLE_BUCKET=<your-bucket-name>

Run integration tests (which hit t3.storage.dev directly):

mix test --include integration