DurableServer

DurableServer provides durable, distributed GenServer processes backed by pluggable storage backends.

It implements fault-tolerant, stateful processes that can survive node failures, restarts, and deployments by automatically persisting state and coordinating across a distributed cluster.

Key Features

Installation

Add durable_server to your list of dependencies in mix.exs:

def deps do

  [

    {:durable_server, "~> 0.1.0"}

  ]

end

For releases, add :os_mon to extra_applications:

def application do

  [

    mod: {MyApp.Application, []},

    extra_applications: [:logger, :runtime_tools, :os_mon]

  ]

end

Basic Usage

defmodule MyCounterServer do

  use DurableServer, vsn: 1



  def dump_state(state), do: %{count: state.count}



  def load_state(_old_vsn, %{"count" => count}), do: %{count: count}



  def init(%{count: count} = state) do

    {:ok, Map.merge(state, %{started_at: DateTime.utc_now()})}

  end



  def handle_call(:increment, _from, state) do

    new_state = %{state | count: state.count + 1}

    {:reply, new_state.count, new_state}

  end



  def handle_call(:get_count, _from, state) do

    {:reply, state.count, state}

  end

end

Start the supervisor (typically in your application supervision tree):

children = [

  {DurableServer.Supervisor,

   name: MyDurableSup,

   prefix: "my_app/",

   object_store: [

     bucket: "my-bucket",

     access_key_id: System.fetch_env!("DURABLE_AWS_ACCESS_KEY_ID"),

     secret_access_key: System.fetch_env!("DURABLE_AWS_SECRET_ACCESS_KEY"),

     s3_endpoint: System.fetch_env!("DURABLE_AWS_ENDPOINT_URL_S3"),

     default_region: System.fetch_env!("DURABLE_AWS_REGION")

   ]}

]

Start and use individual servers:

{:ok, {pid, _meta}} = DurableServer.Supervisor.start_child(

  MyDurableSup,

  {MyCounterServer, key: "user_123", initial_state: %{count: 0}}

)



GenServer.call(pid, :increment)  # => 1

GenServer.call(pid, :increment)  # => 2

GenServer.call(pid, :get_count)  # => 2

:initial_state is required and must be a map. On first boot, DurableServer passes it through dump_state/1, the configured backend's encode/decode path, and then load_state/2 before init/1 or init/2. The dumped initial state must therefore be encodable by your configured backend.

Administrative Cordon

Use terminate_and_cordon_child/3 when you need to stop a DurableServer and make it ineligible for any future starts until an operator explicitly clears the cordon.

# Stop a running child and persist status: :cordoned

:ok = DurableServer.Supervisor.terminate_and_cordon_child(MyDurableSup, pid)



# Or cordon by key. If the child is not running, storage is updated directly.

:ok = DurableServer.Supervisor.terminate_and_cordon_child(

  MyDurableSup,

  "user_123",

  timeout: 10_000

)



# Later, allow explicit starts and LifecycleManager recovery again.

:ok = DurableServer.Supervisor.uncordon_child(MyDurableSup, "user_123")

A cordoned object stores status: :cordoned. While cordoned:

When cordoning by key, DurableServer uses the same lock/orphan checks as normal startup before updating storage. A live owner is asked to persist the cordoned status itself; an orphaned owner is only replaced after the lock is expired.

Storage Backends

DurableServer includes two built-in backends:

Object Storage Backend

{DurableServer.Supervisor,

 name: MyDurableSup,

 prefix: "my_app/",

 backend: {DurableServer.Backends.ObjectStore,

  [

    bucket: "my-bucket",

    access_key_id: "...",

    secret_access_key: "...",

    s3_endpoint: "...",

    default_region: "..."

  ]}}

EKV Backend

Start EKV in your application tree (CAS config is required for DurableServer lock semantics):

ekv_config = [

  name: :durable_ekv,

  data_dir: "/path/to/ekv_store",

  cluster_size: 3

]



children = [

  {EKV,

   name: :durable_ekv,

   data_dir: "/data/ekv/durable",

   cluster_size: 3,

   node_id: System.fetch_env!("EKV_NODE_ID")},

  {DurableServer.Supervisor,

   name: MyDurableSup,

   prefix: "my_app/",

   backend: {DurableServer.Backends.EKVStore, ekv_config}}

]

If you use EKV backend, add EKV to your app's dependencies.

Mirror Backend (Object Storage -> EKV)

Use the mirror backend to dual-write while you cut over reads/writes in phases.

See DurableServer.Backends.MirrorStore for usage and an example rollout.

Configuration Options

DurableServer supports these options in the init/1 return tuple:

State Synchronization

State is synchronized to storage in these scenarios:

  1. Manual sync: Return :sync from any callback: {:noreply, state, :sync}. Manual sync, {:sync, metadata}, and the sync: true callback option are strict durability boundaries. The backend retries classified transient failures within its bounded retry policy. If the write still fails, the server exits with a structured {:sync_failed, reason} fatal-exit reason before acknowledging the callback.
  2. Automatic sync: When :auto_sync is enabled, changes sync on callback return. Periodic sync uses the configured :sync_every_ms interval. After the backend exhausts transient retries, automatic and periodic sync log the failure and keep the dirty in-memory state eligible for a later sync. Storage conflicts remain fatal.
  3. Graceful shutdown: State is always synced before termination

Manual synchronization performs storage work inline. A GenServer.call/3 timeout should be long enough to cover the configured backend's retry window. A caller timeout does not cancel storage work already running in the DurableServer process.

Group

Group provides distributed process groups, registry, lifecycle monitoring, and isolated subclusters.

Monitoring Events

Monitor lifecycle events for DurableServers:

# Monitor a specific key

:ok = Group.monitor(MyDurableSup, "user/123")



# Monitor all keys with a prefix

:ok = Group.monitor(MyDurableSup, "user/")



# Monitor all events

:ok = Group.monitor(MyDurableSup, :all)

Monitors receive {:group, events, info} tuples in their mailbox:

def handle_info({:group, events, _info}, state) do

  Enum.each(events, fn

    %Group.Event{type: :registered, key: key, pid: pid, previous_meta: nil} ->

      # A DurableServer started (previous_meta is nil for first registration)

      :ok

    %Group.Event{type: :unregistered, key: key, reason: reason} ->

      # A DurableServer stopped

      :ok

    _ -> :ok

  end)

  {:noreply, state}

end

Event types: :registered, :unregistered, :joined, :left

:registered and :joined events include a previous_meta field (nil for new, old meta for re-register/re-join). Single operations produce one event per tuple; bulk operations (nodedown, process death) batch all events together.

Joining as a Member

Non-DurableServer processes can join keys to be discoverable and receive dispatched messages:

# Join a key (e.g., from a Phoenix Channel)

:ok = Group.join(MyDurableSup, "room/123", %{type: :channel})



# Re-joining updates metadata in place

:ok = Group.join(MyDurableSup, "room/123", %{type: :channel, status: :active})



# Query all members of a key (DurableServers + joined processes)

members = Group.members(MyDurableSup, "room/123")

# => [{#PID<0.150.0>, %{...}}, {#PID<0.200.0>, %{type: :channel, status: :active}}]



# Leave when done (also happens automatically on process death)

:ok = Group.leave(MyDurableSup, "room/123")

Dispatching to Members

Send messages to all members of a key:

# From a DurableServer, broadcast to all connected channels

Group.dispatch(MyDurableSup, state.key, {:new_message, message})

Named Clusters

For advanced use cases, you can create isolated subclusters where only connected nodes receive events:

# Connect this node to a named cluster

:ok = Group.connect(MyDurableSup, :game_servers)



# Join/monitor/dispatch with the cluster: option

:ok = Group.join(MyDurableSup, "room/123", %{}, cluster: :game_servers)

:ok = Group.monitor(MyDurableSup, :all, cluster: :game_servers)

Note: DurableServers always register in the default cluster to ensure global uniqueness. Named clusters are purely for the pub/sub layer.

Monitor vs Join

These are independent - joining does not monitor events, and monitoring does not make you discoverable.

Running Tests

Unit Tests (with LocalStack)

Start LocalStack for S3-compatible storage:

docker run -d --name localstack -p 4566:4566 localstack/localstack

Run the tests:

mix test

Integration Tests (with Tigris)

Set the required environment variables:

Note: You can add these to a gitignored .env in this project and they will be loaded automatically in test_helper.exs

export DURABLE_AWS_ACCESS_KEY_ID=<your-tigris-access-key>

export DURABLE_AWS_SECRET_ACCESS_KEY=<your-tigris-secret-key>

export DURABLE_AWS_ENDPOINT_URL_S3=https://t3.storage.dev

export DURABLE_AWS_ENDPOINT_URL_IAM=https://iam.storage.dev

export DURABLE_AWS_REGION=<your-region>

export DURABLE_BUCKET=<your-bucket-name>

Run integration tests (which hit t3.storage.dev directly):

mix test --include integration