Shigoto 仕事

PostgreSQL-backed background job processing for Erlang. If you know Oban from Elixir, shigoto fills the same role on a plain Erlang/OTP stack.

Shigoto (仕事, "work") is a feature-rich job queue built on PostgreSQL's FOR UPDATE SKIP LOCKED for safe multi-node operation. No Redis or external broker needed — if you have PostgreSQL, you have a job queue.

Features

Core

Scheduling

Reliability

Resilience (seki integration)

Operations

Testing

Quick Start

Add to your deps:

{deps, [
    {shigoto, "~> 1.10"}
]}.

Configure in sys.config. With kura, name the repo; shigoto runs on the minato pool kura_postgres starts for it. repo must equal that pool's name: the repo module, or the repo's pool key when kura's config sets one.

{shigoto, [
    {repo, my_app_repo},
    {queues, [{<<"default">>, 10}, {<<"emails">>, 5}]},
    {poll_interval, 5000},
    {cron, [
        {<<"daily_cleanup">>, <<"0 3 * * *">>, my_cleanup_worker, #{}}
    ]}
]}

Without kura, start a pool of its own and name it with pool:

{ok, _} = shigoto_db:start_pool(my_app_jobs, #{
    host => "localhost", database => "my_app", user => "postgres", password => "postgres"
}).

shigoto creates or updates its tables when it starts, which needs the pool running and the database there. It tries 5 times; if the schema is still not current it starts without its queues, logs shigoto_migration_pending, retries every 30 seconds and starts the queues once the migration succeeds (see the getting-started guide). Meanwhile shigoto:add_queue/2 answers {error, migration_pending} and shigoto:health/0 reports reason => migration_pending. To migrate sooner, for instance right after kura's migrator has created the database, call it yourself and start the queues at once:

ok = shigoto_migration:up(my_app_repo),
_ = shigoto_migrator:retry_now().

retry_now/0 answers not_running when nothing is pending.

Define a worker:

-module(my_email_worker).
-behaviour(shigoto_worker).
-export([perform/1]).

perform(#{<<"to">> := To, <<"subject">> := Subject}) ->
    send_email(To, Subject),
    ok.

Enqueue jobs:

%% Simple insert
shigoto:insert(#{
    worker => my_email_worker,
    args => #{<<"to">> => <<"user@example.com">>, <<"subject">> => <<"Welcome">>}
}).

%% Scheduled for later
shigoto:insert(#{
    worker => my_cleanup_worker,
    scheduled_at => {{2026, 3, 20}, {3, 0, 0}}
}).

%% With priority and queue
shigoto:insert(#{
    worker => my_urgent_worker,
    args => #{},
    priority => 10,
    queue => <<"critical">>
}).

Transactional Enqueue

With {repo, Repo}, shigoto and kura share transactions on the repo's pool, in both directions:

%% A job inserted inside a kura transaction commits or rolls back with it.
my_app_repo:transaction(fun() ->
    {ok, User} = my_app_repo:insert(UserChangeset),
    {ok, _Job} = shigoto:insert(#{
        worker => welcome_email_worker,
        args => #{<<"user_id">> => maps:get(id, User)}
    }),
    User
end).

%% kura writes inside shigoto:transaction/1 join it too.
shigoto:transaction(fun() ->
    {ok, User} = my_app_repo:insert(UserChangeset),
    {ok, _Job} = shigoto:insert(#{worker => welcome_email_worker, args => #{}}),
    User
end).

A raise inside either rolls back both the rows and the jobs. Telemetry for jobs inserted inside shigoto:transaction/1 fires after the commit; inside a kura transaction it fires on insert, before the commit. Use shigoto:transaction/1 as the outer boundary when telemetry has to be commit-accurate.

Jobs inserted inside a kura transaction on a pool of the same name join it whether shigoto is configured with repo or pool. Only the other direction, kura writes joining shigoto:transaction/1, needs repo; with {pool, Pool}, shigoto:transaction/1 covers shigoto's own statements on that pool.

In tests, jobs join kura_sandbox: a job inserted while the test has the repo's sandbox checked out is rolled back at checkin with the test's other writes. See Testing.

Job Lifecycle

available → executing → completed
                     ↘ retryable → executing (claimed again once the backoff is over)
                     ↘ discarded (max attempts reached)
                     ↘ snoozed → available (rescheduled, the attempt given back)

A failed job with attempts left is retryable, with scheduled_at set to the end of its backoff. The queue claims retryable jobs exactly as it claims available ones, so the retry needs nothing but the polling loop.

Jobs can also be cancelled via shigoto:cancel/2 and retried via shigoto:retry/2. Cancelling a job that is running stops its worker process: at once on the node running it, and within heartbeat_interval when the cancel was made on another node.

Worker Callbacks

All optional except perform/1:

Callback Default Description
perform/1 required Execute the job. Return ok, {error, Reason}, or {snooze, Seconds}
max_attempts/0 3 Maximum retry attempts before discarding
queue/0 <<"default">> Default queue name
priority/0 0 Default priority (higher = claimed first)
timeout/0 300000 Execution timeout in milliseconds
unique/0 — Uniqueness constraints
tags/0 [] Default tags for filtering
backoff/2 exponential Custom retry delay: (Attempt, Error) -> Seconds
rate_limit/0 — Seki rate limiter config
concurrency/0 — Max concurrent executions per node (seki bulkhead)
global_concurrency/0 — Max concurrent executions across all nodes
circuit_breaker/0 — Per-worker circuit breaker thresholds
middleware/0 [] Worker-specific middleware chain
on_discard/2 — Called when a job is permanently discarded

Configuration

Option Default Description
repo none kura repo whose minato pool shigoto shares (read before pool)
pool none minato pool name, when not using repo; one of the two is required
queues [{<<"default">>, 10}] Queue names and concurrency limits
poll_interval 5000 Milliseconds between polling
cron [] {Name, Schedule, Worker, Args} tuples
prune_after_days 14 Days to keep completed/discarded jobs
shutdown_timeout 15000 Milliseconds to wait for in-flight jobs
middleware [] Global middleware chain
encryption_key — 32-byte AES-256-GCM key
encryption_keys [] Ordered key list for rotation (newest first)
encryption_write_format v1 Frame written: v1 (what 1.10 writes and reads) or v2 (bound to column and worker; also covers results and batch callback args). Set v2 only once no node runs 1.10 or can be rolled back to it
encryption_read_formats [v1, v2] Frames that are opened
require_encrypted false Refuse to run a job with plaintext args or meta. Needs a key: with true and no key the application does not start
workers undefined Modules allowed to run as workers; undefined allows any module exporting perform/1
instance undefined Name of this installation, giving it its own advisory locks and NOTIFY channel
advisory_lock_timeout 5000 Milliseconds to wait for an advisory lock before giving up
unique_lock_limit 1000 Advisory locks one insert_all/1,2 call may take for its unique jobs (1 to 5000); a longer list answers {error, {too_many_unique_locks, Needed, Limit}}
notify_min_interval 10 Minimum milliseconds between notification-driven polls of one queue
heartbeat_interval 30000 Milliseconds between heartbeats for executing jobs; a job is stale after twice this without one
rescue_interval 60000 Milliseconds between a node's sweeps that rescue stale jobs
unrunnable_snooze_limit 20 Times in a row a job is put back by a node that cannot run it before that fails the attempt
on_dependency_failure run What a waiting job does when a dependency is discarded or cancelled, unless the job sets it: run, cancel or discard
load_shedding — CoDel config map for seki
queue_weights #{} Weighted polling distribution
fair_queues [] Queues using partition-key fair claiming
notifier — LISTEN/NOTIFY connection config: host, port, database, user, password and any other option of a minato connection, such as ssl, auth or timeout. ssl_options need ssl => true
stage_interval 1000 Milliseconds between due-job stager sweeps (needs notifier)

Supervision Tree

shigoto_sup (one_for_one)
  ├─ shigoto_executor_sup    — simple_one_for_one for job execution
  ├─ shigoto_queue_sup       — one gen_server per queue
  │    ├─ shigoto_queue:default
  │    └─ shigoto_queue:emails
  ├─ shigoto_cron            — cron scheduling with leader election
  ├─ shigoto_pruner          — hourly archival and cleanup
  ├─ shigoto_heartbeat       — periodic heartbeat updates
  ├─ shigoto_notifier        — LISTEN/NOTIFY (optional)
  └─ shigoto_stager          — surfaces newly-due jobs, leader election (optional)

Guides

Ecosystem

Requirements

License

MIT