Shigoto 仕事

PostgreSQL-backed background job processing for Erlang. If you know Oban from Elixir, shigoto fills the same role on a plain Erlang/OTP stack.

Shigoto (仕事, "work") is a feature-rich job queue built on PostgreSQL's FOR UPDATE SKIP LOCKED for safe multi-node operation. No Redis or external broker needed — if you have PostgreSQL, you have a job queue.

Features

Core

Scheduling

Reliability

Resilience (seki integration)

Operations

Testing

Quick Start

Add to your deps:

{deps, [
    {shigoto, "~> 1.10"}
]}.

Configure in sys.config. With kura, name the repo; shigoto runs on the minato pool kura_postgres starts for it. repo must equal that pool's name: the repo module, or the repo's pool key when kura's config sets one.

{shigoto, [
    {repo, my_app_repo},
    {queues, [{<<"default">>, 10}, {<<"emails">>, 5}]},
    {poll_interval, 5000},
    {cron, [
        {<<"daily_cleanup">>, <<"0 3 * * *">>, my_cleanup_worker, #{}}
    ]}
]}

Without kura, start a pool of its own and name it with pool:

{ok, _} = shigoto_db:start_pool(my_app_jobs, #{
    host => "localhost", database => "my_app", user => "postgres", password => "postgres"
}).

shigoto creates or updates its tables when it starts, which needs the pool running and the database there. It tries 5 times; if the schema is still not current it starts without its queues, logs shigoto_migration_pending, retries every 30 seconds and starts the queues once the migration succeeds (see the getting-started guide). Meanwhile shigoto:add_queue/2 answers {error, migration_pending} and shigoto:health/0 reports reason => migration_pending. To migrate sooner, for instance right after kura's migrator has created the database, call it yourself and start the queues at once:

ok = shigoto_migration:up(my_app_repo),
_ = shigoto_migrator:retry_now().

retry_now/0 answers not_running when nothing is pending.

Define a worker:

-module(my_email_worker).
-behaviour(shigoto_worker).
-export([perform/1]).

perform(#{<<"to">> := To, <<"subject">> := Subject}) ->
    send_email(To, Subject),
    ok.

Enqueue jobs:

%% Simple insert
shigoto:insert(#{
    worker => my_email_worker,
    args => #{<<"to">> => <<"user@example.com">>, <<"subject">> => <<"Welcome">>}
}).

%% Scheduled for later
shigoto:insert(#{
    worker => my_cleanup_worker,
    scheduled_at => {{2026, 3, 20}, {3, 0, 0}}
}).

%% With priority and queue
shigoto:insert(#{
    worker => my_urgent_worker,
    args => #{},
    priority => 10,
    queue => <<"critical">>
}).

Transactional Enqueue

With {repo, Repo}, shigoto and kura share transactions on the repo's pool, in both directions:

%% A job inserted inside a kura transaction commits or rolls back with it.
my_app_repo:transaction(fun() ->
    {ok, User} = my_app_repo:insert(UserChangeset),
    {ok, _Job} = shigoto:insert(#{
        worker => welcome_email_worker,
        args => #{<<"user_id">> => maps:get(id, User)}
    }),
    User
end).

%% kura writes inside shigoto:transaction/1 join it too.
shigoto:transaction(fun() ->
    {ok, User} = my_app_repo:insert(UserChangeset),
    {ok, _Job} = shigoto:insert(#{worker => welcome_email_worker, args => #{}}),
    User
end).

A raise inside either rolls back both the rows and the jobs. Telemetry for jobs inserted inside shigoto:transaction/1 fires after the commit; inside a kura transaction it fires on insert, before the commit. Use shigoto:transaction/1 as the outer boundary when telemetry has to be commit-accurate.

Jobs inserted inside a kura transaction on a pool of the same name join it whether shigoto is configured with repo or pool. Only the other direction, kura writes joining shigoto:transaction/1, needs repo; with {pool, Pool}, shigoto:transaction/1 covers shigoto's own statements on that pool.

In tests, jobs join kura_sandbox: a job inserted while the test has the repo's sandbox checked out is rolled back at checkin with the test's other writes. See Testing.

Job Lifecycle

available → executing → completed
                     ↘ retryable → available (retry with backoff)
                     ↘ discarded (max attempts reached)
                     ↘ snoozed → available (rescheduled, the attempt given back)

Jobs can also be cancelled via shigoto:cancel/2 and retried via shigoto:retry/2.

Worker Callbacks

All optional except perform/1:

Callback Default Description
perform/1 required Execute the job. Return ok, {error, Reason}, or {snooze, Seconds}
max_attempts/0 3 Maximum retry attempts before discarding
queue/0 <<"default">> Default queue name
priority/0 0 Default priority (higher = claimed first)
timeout/0 300000 Execution timeout in milliseconds
unique/0 — Uniqueness constraints
tags/0 [] Default tags for filtering
backoff/2 exponential Custom retry delay: (Attempt, Error) -> Seconds
rate_limit/0 — Seki rate limiter config
concurrency/0 — Max concurrent executions per node (seki bulkhead)
global_concurrency/0 — Max concurrent executions across all nodes
circuit_breaker/0 — Per-worker circuit breaker thresholds
middleware/0 [] Worker-specific middleware chain
on_discard/2 — Called when a job is permanently discarded

Configuration

Option Default Description
repo none kura repo whose minato pool shigoto shares (read before pool)
pool none minato pool name, when not using repo; one of the two is required
queues [{<<"default">>, 10}] Queue names and concurrency limits
poll_interval 5000 Milliseconds between polling
cron [] {Name, Schedule, Worker, Args} tuples
prune_after_days 14 Days to keep completed/discarded jobs
shutdown_timeout 15000 Milliseconds to wait for in-flight jobs
middleware [] Global middleware chain
encryption_key — 32-byte AES-256-GCM key
encryption_keys [] Ordered key list for rotation (newest first)
heartbeat_interval 30000 Stale job detection interval
load_shedding — CoDel config map for seki
queue_weights #{} Weighted polling distribution
fair_queues [] Queues using partition-key fair claiming
notifier — LISTEN/NOTIFY connection config
stage_interval 1000 Milliseconds between due-job stager sweeps (needs notifier)

Supervision Tree

shigoto_sup (one_for_one)
  ├─ shigoto_executor_sup    — simple_one_for_one for job execution
  ├─ shigoto_queue_sup       — one gen_server per queue
  │    ├─ shigoto_queue:default
  │    └─ shigoto_queue:emails
  ├─ shigoto_cron            — cron scheduling with leader election
  ├─ shigoto_pruner          — hourly archival and cleanup
  ├─ shigoto_heartbeat       — periodic heartbeat updates
  ├─ shigoto_notifier        — LISTEN/NOTIFY (optional)
  └─ shigoto_stager          — surfaces newly-due jobs, leader election (optional)

Guides

Ecosystem

Requirements

License

MIT