⚓ Belay

Durable execution for Elixir — without a workflow server.

Resume at the first unfinished step. Add budgets, signals, DAGs, replay, and an embedded dashboard. Keep it all on Postgres.

Background jobs get awkward when one run contains twelve paid API calls, a human approval, and a fan-out across the cluster. Classic queues retry the whole job. Belay makes the step the unit of recovery: once a step's result commits to the journal, later attempts replay it in microseconds and continue from the first unfinished step. The same journal powers spend accounting and replay debugging.

Dependencies: Postgrex, Jason, telemetry — no Ecto requirement, no Redis, no separate service. Apache-2.0.

Quick start

# mix.exs
{:belay, "~> 1.0.0-rc.5"}

# config/config.exs — the Ecto/Phoenix convention, keyed by the instance name
config :my_app, MyApp.Belay,
  queues: [default: 10, mailers: [limit: 20]],
  crons: [[name: "digest", expr: "0 8 * * 1-5", worker: MyApp.Digest]]

# config/runtime.exs — runtime values (the database URL)
config :my_app, MyApp.Belay,
  storage: [adapter: :postgres, url: System.fetch_env!("DATABASE_URL")]

# application.ex — otp_app pulls the config above; inline opts still override
children = [
  {Belay, otp_app: :my_app, name: MyApp.Belay},
  {Belay.Dashboard, belay: MyApp.Belay, port: 4004}
]

# once, at deploy time (idempotent)
Belay.Storage.Postgres.migrate!(db_url)

Prefer everything inline in the supervision tree? That works too — pass the same keys to {Belay, name: ..., storage: ..., queues: ...} and skip otp_app.

defmodule MyApp.WelcomeEmail do
  use Belay.Worker, queue: :mailers, max_attempts: 5

  @impl Belay.Worker
  def run(ctx) do
    MyApp.Mailer.deliver_welcome(ctx.job.input["user_id"])
  end
end

Belay.insert(MyApp.Belay, MyApp.WelcomeEmail.new(%{"user_id" => 42}))

Testing is deterministic — drain synchronously, travel through time:

{:ok, job} = Belay.insert(name, MyApp.Pipeline.new(%{"id" => 1}))
assert %{succeeded: 1} = Belay.Testing.drain(name, :default)

Belay.Clock.Sim.advance(clock, 3_600)   # backoff, cron, rate windows, leases…

Steps, waits, fan-out — one worker

Where Belay pulls ahead of a classic queue is jobs with expensive parts, long waits, and moving pieces:

defmodule MyApp.ResearchPipeline do
  use Belay.Worker, queue: :pipelines, max_attempts: 10

  @impl Belay.Worker
  def run(ctx) do
    # Committed steps replay from the journal; unfinished steps run again.
    text    = Belay.step(ctx, :transcribe, fn -> whisper!(ctx.job.input["url"]) end)
    summary = Belay.step(ctx, :summarize, fn -> llm!(text) end, cost: [usd: 0.02, tokens: 1200])

    # Fan out real jobs across the cluster; park at zero cost until all land.
    checks = Belay.map_children(ctx, :verify, MyApp.FactCheck,
               Enum.map(summary.claims, &%{"claim" => &1}))

    # Human in the loop: durable wait, instant wake on signal.
    case Belay.await(ctx, :approval, timeout: 86_400) do
      %{"approved" => true} -> {:ok, %{summary: summary, checks: checks}}
      _ -> {:cancel, :rejected}
    end
  end
end

# Attach a job budget and enforce uniqueness at insert:
Belay.insert(MyApp.Belay,
  MyApp.ResearchPipeline.new(%{"url" => url}, budget: [usd: 1.00], unique: "research:#{url}"))

Step bodies have at-least-once execution until their result is committed. If a process dies after an external side effect but before the journal write, that body can run again. Use provider idempotency keys or make external writes idempotent. Once the journal row exists, normal retries do not re-execute it.

The same engine runs the unglamorous fleet — emails, webhooks, image processing, nightly crons — and the newer shapes those queues were never built for: declared workflow DAGs, month-long approval flows parked at zero cost, and LLM/agent pipelines where each API call is a journaled step with a price on it.

Budgets and spend control

LLM-heavy workloads add a failure mode classic queues don't model: cost. A retry that re-runs completed API calls buys them twice; a usage spike outruns any dashboard. Belay enforces spend in the engine, in three layers:

# 1. Per-job fail-after-crossing limit. The step that crosses the limit runs;
#    no later unfinished step is admitted. Overshoot is bounded by that
#    step's declared cost in the modeled single-attempt path.
MyApp.ResearchPipeline.new(input, budget: [usd: 1.00])

# 2. Fleet-wide admission budget per window — a resource bucket spans every
#    queue that names it. Claims debit estimates atomically across the fleet.
queues: [
  ai: [limit: 20,
       rate: [resource: "anthropic_tokens", allowed: 2_000_000, period: 60, estimate: 3_000]]
]
# When the estimated allowance is spent, claims stop and work queues.

# 3. True-up — the claim debits the estimate; inside the job you correct it
#    with actuals, so the window converges on what the invoice will say.
#    A job debits its own queue's resource:
Belay.step(ctx, :summarize, fn ->
  %{text: text, usage: usage} = llm!(prompt)
  Belay.debit(ctx, "anthropic_tokens", usage.total_tokens)
  text
end, cost: [usd: 0.02])

Committed steps replay from the journal at zero declared cost, and budget checks read durable spend, so a later attempt cannot forget recorded cost. As with any at-least-once system, an external charge made before a crash but not journaled may repeat and is not visible to Belay; use the provider's idempotency mechanism for that window.

Why it's built this way

• The journal is the core. Steps, costs, events, and results are first-class columns — so budgets, token-true-up rate limits, streaming, and replay debugging fall out of the schema instead of being bolted on.
• Leaderless everything. No peer election exists in the codebase: cron dedupes through a unique index, recovery is idempotent row-level work any node performs. The "leader stalled, nothing runs" failure class is structurally absent.
• Millisecond dispatch on a polling-floor guarantee. Adaptive burst polling plus an opt-in pg_notify accelerator — measured 11ms p50 insert→result across unconnected processes — while nothing load-bearing touches LISTEN/NOTIFY, so PgBouncer transaction pooling and serverless Postgres just work.
• Leases with fencing, not rescue heuristics. Crashed workers' jobs are reclaimed in seconds; zombie acks are rejected by attempt fencing.
• Deterministic by design. Storage behind a behaviour with an in-memory reference adapter; the clock injectable everywhere (SQL included). One suite runs against both adapters, and time-dependent behavior is tested by time travel, never sleeps.

The workflow DAG view

Workflows, fan-outs, and spawned children render as a live graph — deep-linkable (#workflow=<id>), with the full step journal one click away:

Feature tour

Measured, not claimed

Numbers from this repo's reproducible harnesses on a laptop (Postgres 16):

Six real bugs were found by these harnesses before any user could: two by the chaos soak (a READ COMMITTED wake race among them), one by the 7-hour endurance run (the budget crash window), one by adapter-equivalence property testing (lost cancel requests), and two by extending the formal model (retry semantics). The model is validated by rediscovery: revert any fix in it and TLC reproduces the production failure, step for step (CHANGELOG, verify/). The race lessons are codified in the wire contract.

Proven on a real app

The first design-partner port replaced Oban in a production-shaped Phoenix ingestion service (4 workers, webhooks, GDPR deletion flows, hourly cron, uniqueness everywhere): 222/222 tests green and a live-producer run on the first attempt, with zero engine changes required. Dead-node recovery tightened from a conservative 45-minute rescue plugin to ~2× a 60-second lease — renewed leases replace guessing.

Coming from Oban?

Workers port mechanically (perform/1 → run/1), the plugins you delete are engine config (Pruner → retention:, Lifeline → renewed leases, Cron → crons:), and pending jobs move in one command:

mix belay.migrate_oban --url postgres://.../my_app            # dry-run report
mix belay.migrate_oban --url postgres://.../my_app --execute

The migration guide has the full porting map, the uniqueness translation, and the archive pattern for historical rows — with field notes from a real port that went 222/222 on the first run.

Docs

Getting started · Migrating from Oban · Durable steps · Building agents · Operations · Testing · Honest comparison · Formal verification · Architecture · Wire contract

Status

1.0.0-rc.5. The API is broad and the evidence is unusually deep for a new engine, but it is still a release candidate. Production miles are the one feature that cannot be compressed into a test suite; evaluate it against your failure modes before replacing a queue that already works. Post-1.0 candidates: SQLite storage, batched acking, and additional language clients.

License

Apache-2.0. Contributions welcome — CONTRIBUTING.md's five rules are the soul of the codebase.