ObanSentinel

ObanSentinel is a telemetry-driven circuit breaker and execution guard for Oban. It pauses queues when a worker's rolling failure rate indicates a dependency or code-path failure, then safely recovers the queue after a cooldown. Worker guards prevent unbounded job duration and process memory growth.

Installation

Add the dependency:

def deps do
  [{:oban_sentinel, "~> 0.1.0"}]
end

See the complete configuration reference for every plugin, policy, and webhook option.

Req is optional and only required for the built-in webhook notifier:

{:req, "~> 0.5"}

Oban configuration

Add Sentinel to the same configuration that starts your named Oban instance:

# config/config.exs
config :my_app, Oban,
  repo: MyApp.Repo,
  queues: [critical: 10, default: 20],
  plugins: [
    {ObanSentinel.Plugin,
     # `lookback` / `cooldown` are friendly aliases for the explicit *_ms keys.
     lookback: :timer.minutes(5),
     failure_threshold: 0.25,
     minimum_samples: 20,
     cooldown: :timer.minutes(5),
     recovery_check_interval_ms: :timer.seconds(30),
     # Set :emit_only first to observe breaches without pausing queues.
     action: :pause_queue,
     auto_recover: true,
     max_samples_per_worker: 10_000,
     notifier: ObanSentinel.Notifier.Webhook,
     notifier_opts: [url: System.fetch_env!("SENTINEL_WEBHOOK_URL")]}
  ]

When using a named instance, Oban supplies :oban to plugins. The breaker tracks each worker independently but pauses the worker's configured queue; therefore workers with different risk profiles should use separate queues.

Guarding a worker

The guard deliberately requires perform_guarded/1: macro expansion cannot safely replace an already-defined perform/1 in every Elixir compilation order. perform/1 remains the normal Oban callback generated by the guard.

defmodule MyApp.ExternalApiWorker do
  use Oban.Worker, queue: :critical, max_attempts: 8
  use ObanSentinel.Guard,
    max_duration: :timer.seconds(20),
    max_memory_mb: 80

  @impl Oban.Worker
  def perform_guarded(%Oban.Job{args: %{"id" => id}}) do
    MyApp.ExternalApi.sync(id)
  end
end

Duration and memory breaches return {:error, {:oban_sentinel, reason}} to Oban, which follows the worker's ordinary retry policy. The memory guard polls the isolated task at 25 ms intervals. It bounds the task's BEAM heap; it is not a substitute for OS/container memory limits or for validating untrusted code.

Telemetry

Sentinel consumes Oban :start, :stop, and :exception job telemetry. It emits these events:

• [:oban_sentinel, :circuit_breaker, :tripped]
• [:oban_sentinel, :circuit_breaker, :recovered]

Both emit metadata containing :worker, :queue, and the relevant failure/cooldown fields. A :recovered event emits %{}; a :tripped event emits %{failure_rate: float(), queue_action: :ok | :skipped | :deferred | {:error, term()}}. Notification delivery runs in a separate task and never blocks Oban telemetry handlers.

Inspection and dry runs

Use snapshot/2 to expose Sentinel state through your own authenticated admin surface or metrics exporter. The ETS table remains private to the breaker process, so callers never need direct table access.

ObanSentinel.CircuitBreaker.snapshot(MyApp.ObanSentinelBreaker, MyApp.Workers.ApiSync)
# => %{status: :open, samples: 25, failures: 12, failure_rate: 0.48,
#      starts: 25, duration: %{count: 25, average: 3_200_000, max: 9_000_000}}

For an authenticated operations surface, enumerate observed workers with ObanSentinel.CircuitBreaker.workers/1. On-call operators can close a circuit early with reset/3; it resumes the queue only when no other Sentinel circuit for that queue remains open.

:ok = ObanSentinel.CircuitBreaker.reset(MyApp.ObanSentinelBreaker, MyApp.Workers.ApiSync)

Telemetry durations use native :telemetry units. Avoid displaying them as milliseconds unless your exporter explicitly converts them.

Guard limit breaches emit [:oban_sentinel, :guard, :duration_exceeded] or [:oban_sentinel, :guard, :memory_exceeded]. Their metadata contains the active limit in milliseconds (:max_duration) or bytes (:max_memory).

Set action: :emit_only during rollout to emit Sentinel events without calling Oban.pause_queue/2 or Oban.resume_queue/2. This is the recommended first production deployment mode.

Per-worker policies

Set stricter or more cautious policies for individual workers without running another plugin. Oban reports worker names as strings, so policy keys should use the worker's fully-qualified module name.

policies: %{
  "MyApp.Workers.PaymentCapture" => %{
    failure_threshold: 0.10,
    minimum_samples: 5,
    cooldown_ms: :timer.minutes(10)
  },
  "MyApp.Workers.ExperimentalImport" => %{
    action: :emit_only,
    auto_recover: false
  }
}

Policy fields inherit the plugin defaults when omitted. Keep workers with different queue-control policies in separate queues: Oban’s pause/resume operation controls the entire queue.

Operational notes

• A resumed queue may retrip immediately if failures continue; that is the expected protective behavior.
• Manual queue pauses are indistinguishable from Sentinel's pause to Oban. Do not enable automatic recovery for queues you pause manually for maintenance.
• Set a sensible minimum_samples so a few transient failures cannot trip a production queue.
• Sentinel can only safely resume queues it paused itself. If your operational process manually pauses queues, set auto_recover: false for that instance.
• Memory guards observe the isolated task's BEAM memory. They cannot constrain native allocations, ports, or memory consumed by external programs.

Development

mix deps.get
mix format --check-formatted
mix test

See ARCHITECTURE.md for event flow and state ownership, and SECURITY.md for operational boundaries and deployment guidance.