HoneydewEctoNotifyQueue

Table of Contents

• HoneydewEctoNotifyQueue
  • Description
  • Notes
  • Setting it up
  • Running the tests
  • Custom job configuration persistence

Description

HoneydewEctoNotifyQueue is a queue built for Honeydew that uses postgres notifications instead of polling. It was originally built before honeydew offered the ecto polling solution.

Contrary to the honeydew ecto polling solution, this package sets up two independent tables for managing the queue of jobs, and managing configuration over multiple instances;

A jobs table tracks jobs and job_configs table tracks configurations.

Notes

Honeydew.yield is not supported by this adapter.

Jobs are reserved using postgres' FOR UPDATE NOWAIT locking.

It is possible to suspend all job processing across instances by updating the suspended job config;

{:ok, _config} = HoneydewEctoNotifyQueue.Config.update_config(MyApp.Repo, "suspended", "true")

See more about configuration handling here

Setting it up

Note: You should read how to install honeydew here first

This queue takes some additional options. An example below,

import Supervisor.Spec

def background_job_processes do
  [
    notifier_process(),
    Honeydew.queue_spec(:process, # queue_name
      queue: {HoneydewEctoNotifyQueue, [
                  repo: YourApp.Repo, # your app's Repo module
                  max_job_time: 3_600, # seconds
                  retry_seconds: 15, # seconds,
                  notifier: YourApp.Notifier # this should match the `name:` in `notifier_process` below
                ]},
      failure_mode: {Honeydew.FailureMode.Retry, times: 3}
    ),
    Honeydew.worker_spec(:process, YourApp.Worker, num: 1)
  ]
end

def notifier_process do
  worker(Postgrex.Notifications, [YourApp.Repo.config() ++ [name: YourApp.Notifier]])
end

def start(_type, _args) do
  children = [
    # ... The rest of your app's supervision tree
  ] ++ background_job_processes
  
  Supervisor.start_link(children, opts)
end

Running the tests

$ MIX_ENV=test mix do ecto.create, ecto.migrate
$ mix test

Custom job configuration persistence

This queue also adds support for persisting job configuration state via postgres. By default, this is how queue suspension is managed across multiple instances.

You can leverage the existing notification setup to synchronise other configurations across instances.

An example of this may be the disabling of automatic queuing of a job when an API is hit.

You can see an example of how to listen for configuration changes in examples/configuration_listener.ex