Trellis for Elixir

Embedded Trellis for Elixir apps: a Rustler NIF over the trellis crate's BlockingTrellis, so a Phoenix app can define and run streaming transforms without deploying a separate service. The design is ADR-0010.

Installation

The Hex package is trellis_pg (Hex's trellis is taken); the modules are Trellis.*, and the OTP application is :trellis_pg.

def deps do
  [{:trellis_pg, "~> 0.2.0"}]
end

Installing needs no Rust: compiling the package downloads the NIF built for the machine from the release's GitHub page and checks it against the SHA-256 the package ships. Builds exist for

On any other target, or to build it yourself, build the NIF from source. It needs a Rust toolchain (the version in the repository's rust-toolchain.toml) and Rustler in the host's own deps:

{:rustler, "~> 0.38.0", runtime: false}
TRELLIS_PG_BUILD=1 mix deps.compile trellis_pg --force

The choice is made when trellis_pg compiles, so --force it after changing the variable. config :rustler_precompiled, :force_build, trellis_pg: true does the same from config. Cargo fetches the trellis crate from this repository at the released commit.

The NIF is built without the trellis crate's otlp feature, so neither path pulls in an OpenTelemetry or gRPC stack. The engine's log lines reach Logger through Trellis.LogBridge.

Usage

The surface mirrors the Rust crate's BlockingTrellis. Every function has a bang variant.

# A deploy's migration step: the defaults run nothing in the background.
{:ok, migrator} = Trellis.connect(url: "postgres://localhost/app")
# TLS options go in the URL (docs/embedding.md, "TLS to Postgres"):
#   url: "postgres://db.example.com/app?sslmode=verify-full&sslrootcert=/etc/ssl/ca.pem"
:ok = Trellis.migrate(migrator)
:ok = Trellis.shutdown(migrator)

# The app, once migrated. The staging worker reads the catalog as it starts,
# so a `staging: true` handle can't connect before `migrate/1` has run.
{:ok, trellis} = Trellis.connect(url: "postgres://localhost/app", staging: true, drain_threads: 2)
{:ok, _definition} = Trellis.define(trellis, "TRANSFORM widget_prices FROM widgets SELECT price AS price")
{:ok, %Trellis.Status{status: :live}} = Trellis.status(trellis, "widget_prices")

# A column that paused on bad data: find the rows, fix them, resume it.
{:ok, [%Trellis.QuarantineEntry{target: "widget_prices.price"} | _]} = Trellis.quarantined(trellis)
{:ok, %Trellis.SamplePage{samples: rows, next_cursor: cursor}} =
  Trellis.sample_quarantined(trellis, "widget_prices.price", limit: 50)
{:ok, {:resumed, ["widget_prices.price"]}} =
  Trellis.apply(trellis, "RESUME TRANSFORM widget_prices.price")

:ok = Trellis.shutdown(trellis)

connect/1's defaults (staging: false, drain_threads: 0) run nothing in the background. For each instance (catalog schema), some connection in the fleet must set staging: true and some connection must run drain threads, or no definition ever reaches :live. Any number of connections may set staging: true: one takes the producer lock and leads, the rest stand by and take over when it goes away, so a rolling deploy needs no operator step and the application boots even when the old worker is still running (see Rolling deploys).

Resources

At the defaults, a handle holds up to 20 Postgres connections (its call pool, opened as calls need them and closed after 60 seconds idle) and 3 threads. A worker handle (staging: true, drain_threads: N) also owns a background pool of up to 2N + 6 connections (2N + 5 without staging, 2 with staging alone), closed after 60 seconds idle, and two connections that stay open: the staging session and one LISTEN shared by its drain threads. It holds 6 threads: drain threads are tasks on its runtime, not OS threads. Its memory is bounded by row counts, not bytes: about N × 100,000 changed rows' images at its peak. :max_call_pool_size lowers the 20 (the background pool is sized by the handle's roles, not by an option), and :worker_threads sets the threads. Every handle (each {Trellis, name: ...} child) costs this again. The full accounting is in docs/embedding.md.

Conventions

In a Phoenix app

Let the application's supervisor own the handle. {Trellis, options} starts a process that connects with connect/1's options as it starts, hands the handle to the callers that make calls through its name, and shuts the handle down when the supervisor stops it:

# config/runtime.exs
database_url = System.fetch_env!("DATABASE_URL")

config :my_app, MyApp.Trellis,
  name: MyApp.Trellis,
  url: database_url,
  # Only the fleet's one worker node runs the background work.
  staging: System.get_env("TRELLIS_WORKER") == "true",
  drain_threads: if(System.get_env("TRELLIS_WORKER") == "true", do: 2, else: 0)

# lib/my_app/application.ex
children = [
  MyApp.Repo,
  {Trellis, Application.fetch_env!(:my_app, MyApp.Trellis)},
  MyAppWeb.Endpoint
]

# Anywhere in the app: the name stands in for the handle.
{:ok, status} = Trellis.status(MyApp.Trellis, "widget_prices")

Each call runs in the process that makes it, in parallel with every other, so no process queues behind another's call. The caller has to be on the same node: a handle works only on the node that connected it. shutdown/1 refuses the name; the supervisor stops it, which cancels the calls still in flight.

Define transforms in Ecto migrations with Trellis.Migration. Trellis never joins a migration's transaction, so a migration that uses it must set @disable_ddl_transaction true (the helpers raise before applying anything if a transaction is open), and must define up/0 and down/0:

defmodule MyApp.Repo.Migrations.DefineWidgetPrices do
  use Ecto.Migration
  use Trellis.Migration

  @disable_ddl_transaction true

  def up do
    define "TRANSFORM widget_prices FROM widgets SELECT price AS price"
  end

  def down do
    apply "PAUSE TRANSFORM widget_prices"
    apply "DROP TRANSFORM widget_prices"
  end
end

Ecto doesn't start the application to migrate, so each helper connects a handle of its own, from the repo's :trellis configuration (config :my_app, MyApp.Repo, trellis: [url: database_url]), and runs migrate/1 on it first. A define isn't idempotent, and the helpers don't make it so: Ecto's schema_migrations runs each migration once. See Trellis.Migration for the details, and for an optional guard.

Add import_deps: [:trellis_pg] to the app's .formatter.exs to keep define and apply free of parentheses, like Ecto's own commands.

Fixtures and bulk loads in tests

A loader that disables triggers for the load (ALTER TABLE ... DISABLE TRIGGER ALL, pg_restore --disable-triggers) disables Trellis's capture triggers too, and ENABLE TRIGGER ALL leaves them in a mode Trellis treats as a disruption. The rows it wrote never reached a target, so Trellis pauses every definition that reads the table (capture_failure has kind: :disruption) instead of letting a test read silently wrong data, and waits for a RESUME. Trellis.TestSupport does that RESUME for a suite, then waits for the rebuilt targets to catch up:

setup do
  # ... load the fixtures with triggers disabled, then re-enabled ...
  {:ok, _resumed} = Trellis.TestSupport.resume_capture_disruptions(trellis)
  :ok
end

resume_capture_disruptions/2 first pauses the readers of any disrupted table itself (Trellis.detect_capture_disruptions/1), so it doesn't wait for the staging worker's next pass; resumes only the definitions paused with kind :disruption; waits until each is :live and every write committed so far has reached its targets; and returns {:ok, resumed} ([] when nothing was disrupted, which costs one catalog read per captured table). A transform an operator paused, or one paused by a schema change or another failure, stays paused. A RESUME the engine refuses (a table that is gone, a table define would now refuse) leaves that definition paused; the rest are still resumed, and the call then returns {:error, {:resume_refused, refused}}, a map of target to reason. resume_capture_disruptions!/2 returns the targets and raises Trellis.TestSupport.ResumeRefused (or Trellis.Error). The :timeout_ms option (default 30 000) bounds the wait. The handle needs its staging worker and a drain worker (staging: true, drain_threads: 1), because the resume rebuilds through them.

Trellis sees a write when it commits, so with Ecto's SQL sandbox, load the fixtures that targets must reflect outside the sandbox transaction (in setup_all, or with the sandbox in :auto mode) and call the helper once.

Schema dumps

mix trellis.dump writes the structure file mix ecto.dump writes, with the definitions in it the way Ecto puts schema_migrations in it, and mix trellis.load loads it. Make them what mix ecto.dump and mix ecto.load run, by aliases in mix.exs:

defp aliases do
  ["ecto.dump": "trellis.dump", "ecto.load": "trellis.load"]
end

The file has the application's tables, the target tables (with their indexes and grants), the schema_migrations rows, and one row per definition: its source, target and JOIN table, its text as written, its grammar version and its floor. Trellis's own schema, the capture triggers on source tables and the 1-1 targets' ledgers are not in it, so regenerating the file changes it only where a definition changed. mix trellis.load runs mix ecto.load and then Trellis.migrate/1 for each repo with a :trellis configuration, which creates Trellis's tables and restores each definition into its dumped target. Each is restored unbuilt, and the worker builds it from the source rows; a paused or quarantined transform is restored unpaused, because a pause is runtime state. A file loaded another way (psql -f) needs Trellis.migrate/1 after it. How the restore works, and what it refuses, is in docs/embedding.md.

mix ecto.dump can't pass pg_dump extra arguments, so for a repo with a :trellis configuration (as for Trellis.Migration) mix trellis.dump runs pg_dump itself, with ecto.dump's arguments (-r, -d, --prefix, -q) and Trellis.schema_dump/1's, which leave Trellis's schema and the ledgers out. With --prefix, the instance's target schema is dumped too. It takes the capture triggers out of the output (Trellis.filter_schema_dump/2), and reads the definitions before and after the pg_dump, redoing the dump, up to three times, when a define or drop committed in between, so the file's DDL and definitions agree. A repo with no :trellis configuration is dumped by mix ecto.dump.

mix trellis.dump also dumps a repo's :migration_repo, as mix ecto.dump does, before the Trellis repo's file, and with --prefix it reads schema_migrations from the schemas named only (not the target schema it adds to the dump). The repo and its :trellis url have to name the same database, by host, port and database name spelled alike (localhost is not 127.0.0.1; a repo connecting over a socket directory, or naming no host, matches a local host), or mix trellis.dump and mix trellis.load refuse, before anything is written or loaded. A dump that fails once pg_dump has written the file removes it, since its raw pg_dump output would load without the definitions; a file no pg_dump wrote to is left as it was. mix trellis.load restores nothing for a repo mix ecto.load left alone (--skip-if-loaded on a loaded database, or a "no" to its prompt).

pg_dump 17 writes CREATE SCHEMA public; when it dumps public by name, which stops mix ecto.load into a database mix ecto.create made; the task writes CREATE SCHEMA IF NOT EXISTS public; instead.

A structure file written by an earlier release, which carries the Trellis schema and the definitions' catalog rows, still loads. Trellis.StructureDump.append/2 writes that format onto a structure file someone else dumped with the Trellis schema in it (reading the definitions once, so it can't see a define racing that dump), and Trellis.dump_definitions_sql/1 returns its SQL.

Layout

Development

The Erlang and Elixir versions are pinned in .tool-versions. From this directory:

cargo build -p testkit --bin trellis-testkit   # the suite's throwaway Postgres
mix deps.get
mix test

In this repository (:dev and :test) the NIF always builds from source, never downloads. mix test compiles it through Cargo into the workspace's target/, and the suite spawns trellis-testkit (from PATH, else the workspace's debug build) for a Postgres cluster that is torn down when the test run exits. It needs the Postgres server binaries on PATH, like the Rust tests.

test/trellis/parity_test.exs runs clients/parity/cases.json, the fixture the Ruby suite runs too, so the two bindings can't drift apart unnoticed (see clients/parity/README.md).

Releasing

trellis_pg is released with the rest of Trellis, from one v<version> tag (docs/releasing.md). @version in mix.exs and the NIF crate's version (native/trellis_nif/Cargo.toml) are the release's. .github/workflows/release.yml builds the NIF for each target above and attaches them to the tag's GitHub release, where lib/trellis/native.ex downloads them from. The package is then published to Hex by hand with .github/scripts/publish-packages.sh, which writes checksum-Elixir.Trellis.Native.exs from the NIFs that release serves.

.github/scripts/build-elixir-nif.sh <target> <out-dir> builds one NIF the way the workflow does. Linux targets need cargo-zigbuild and Zig (pip install cargo-zigbuild ziglang).