JoobQ SDK for Elixir

A client for the JoobQ SDK gateway of a self-hosted JoobQUI server: enqueue jobs now, later or at a given time, with deadlines, idempotency keys and fields the gateway seals, one at a time (JoobQ.enqueue/4, enqueue_async/4) or up to 500 in one call (enqueue_many/2), and run them in the application with use JoobQ.Worker and the instance's :queues and :workers options (JoobQ.Config); register job schedules, use JoobQ Pro batches and rate limits, and start and talk to workflow runs, and build workflows whose steps the worker runs (JoobQ.Workflow, the instance's :workflows option). JoobQUI version 0.8.0; an SDK talks to a server of its own version or later.

Workers

children = [
  {JoobQ,
   url: "https://joobqui.example.com",
   token: System.fetch_env!("JOOBQ_SDK_TOKEN"),
   queues: ["mail"],
   workers: [MyApp.Welcome],
   middleware: [MyApp.Timing]}
]

Schedules

JoobQ.register_schedule("nightly", "mail", "digest", cron: "0 3 * * *") registers a job schedule, or updates it: it is idempotent by its name (:every takes "30 seconds", "5 minutes", ...; see JoobQ.Schedules). It fires on the server; editing, pausing and deleting schedules are done in the console.

Workflows (JoobQ Pro)

The joobq-pro engine conducts every run of a registered blueprint: it retries, waits, compensates and keeps the run's state; a worker only runs steps. JoobQ.Workflow builds and registers workflows, JoobQ.Workflows starts runs and talks to them.

import JoobQ.Workflow

checkout =
  new("checkout", task_queue: "workflows")
  |> step("reserve", &Shop.reserve/2,
    input: %{sku: input("sku")},
    retry: retry(attempts: 3, non_retryable: [Shop.OutOfStock]),
    compensate: compensation(&Shop.release/2))
  |> parallel(fn wf -> wf |> step("charge", &Shop.charge/2) |> step("receipt", &Shop.receipt/2, after: ["charge"]) end)
  |> wait_for("approval", signal: "approve", timeout_ms: 60_000)
  |> query("total", output("charge", "amount"))

children = [{JoobQ, url: url, token: token, workflows: [checkout]}]

From a producer: JoobQ.start_workflow(name, input, id: "order-1"), signal_workflow/4, update_workflow/4, query_workflow/3, upsert_workflow_attributes/2, and the reads describe_workflow/2, workflow_result/2 and workflow_history/2 (JoobQ.Workflows). Each is sent once; a run id is one percent-encoded path segment and stays out of telemetry.

Batches (JoobQ Pro)

JoobQ.create_batch(jobs, on_complete: {"mail", "report"}, context: %{"tenant" => "t1"}) creates, fills and seals a batch of 1 to 1000 jobs in one call; JoobQ.get_batch(id) reads its progress. The gateway enqueues the callbacks (:on_failure at the first dead member, :on_complete when all are counted, :on_success then too when none is dead), each exactly once; the SDK never does. See JoobQ.Batches for what a member and a callback may carry.

Rate limits (JoobQ Pro)

JoobQ.rate_limit("tenant-1", 10, 60_000) reserves one of 10 slots a minute for the key "tenant-1" on the gateway, in one step: {:ok, %{"allowed" => true}}, or "allowed" => false with "retry_after_ms". It paces your own code and never waits; without a Pro licence it answers {:error, %JoobQ.Error{reason: :pro_required}}.

Telemetry

Spans (:start, then :stop or :exception, with duration in native units; see JoobQ.Telemetry):

Event Metadata :stop adds
[:joobq, :enqueue, *] instance, queue, job_type result, status, jid, reason
[:joobq, :schedule, *] instance, schedule, queue, job_type result, reason
[:joobq, :rate_limit, *] instance, limit, period_ms (never the key) result, allowed, reason
[:joobq, :batch, *] instance, size (create_batch/2; get_batch/2 has only :request) result, reason, batch_id
[:joobq, :job, *] instance, queue, job_type, jid, attempt, task_kind outcome, retryable, error_class
[:joobq, :workflow, *] instance, call (:start, :signal, ...), name result, reason
[:joobq, :blueprint, *] instance, workflow result (:registered, :refused, :retry)
[:joobq, :request, *] name, method, path status

A workflow step's job span is a job's: it adds only task_kind ("activity", "compensation", "update"; "job" for a job), and its job_type is the activity name ("<workflow>/<step>").

:exception adds kind and error_class only. None of JoobQ's own events carries a payload, an output, a lease token or the SDK token. Finch's own [:finch, ...] events carry the request (the token in its headers, the body): do not forward those to a sink you do not trust.

Raising variants

enqueue!/4 and enqueue_many!/2 raise the error instead of answering it; the other calls (schedules, rate limits, batches, workflows) answer {:ok, answer} or {:error, %JoobQ.Error{}} only.

Retries and limits

A producer call is sent once and never retried. An error with reason :transport (no answer) or :bad_response (a 2xx answer that is not JSON) MAY mean the job was stored: the gateway can store it before the connection breaks, and it does not deduplicate by jid. Give a job a unique_key to make sending it again safe.

More about JoobQ: https://www.joobq.io

License

MIT (see LICENSE).