Trellis for Elixir
Embedded Trellis for Elixir apps: a Rustler
NIF over the trellis crate's BlockingTrellis, so a Phoenix app can define
and run streaming transforms without deploying a separate service. The design
is ADR-0010.
Installation
The Hex package is trellis_pg (Hex's trellis is taken); the modules are
Trellis.*, and the OTP application is :trellis_pg.
def deps do
[{:trellis_pg, "~> 0.2.0"}]
end
Installing needs no Rust: compiling the package downloads the NIF built for the machine from the release's GitHub page and checks it against the SHA-256 the package ships. Builds exist for
- Linux, glibc 2.17 or newer:
x86_64andaarch64 - Linux, musl (Alpine):
x86_64 - macOS: Apple silicon and Intel
On any other target, or to build it yourself, build the NIF from source. It
needs a Rust toolchain (the version in the repository's rust-toolchain.toml)
and Rustler in the host's own deps:
{:rustler, "~> 0.38.0", runtime: false}
TRELLIS_PG_BUILD=1 mix deps.compile trellis_pg --force
The choice is made when trellis_pg compiles, so --force it after changing
the variable. config :rustler_precompiled, :force_build, trellis_pg: true
does the same from config. Cargo fetches the trellis crate from this
repository at the released commit.
The NIF is built without the trellis crate's otlp feature, so neither
path pulls in an OpenTelemetry or gRPC stack. The engine's log lines reach
Logger through Trellis.LogBridge.
Usage
The surface mirrors the Rust crate's BlockingTrellis. Every function has a bang variant.
connect/1,migrate/1,config/1,shutdown/1: the handle's lifecycle.apply/2: any statement of Trellis's grammar (TRANSFORM,PAUSE TRANSFORM,RESUME TRANSFORM,DROP TRANSFORM,ALTER TRANSFORM), returning what it did as a tagged value ({:transform_defined, target_table},:paused,{:resumed, addresses}, ...).define/2is anapply/2that refuses anything but aTRANSFORM.status/2,definitions/1: the catalog reads.quarantined/1,quarantine_status/2,sample_quarantined/3,poisoned_since/2: the quarantine reads. Resuming is a statement.release_key/4releases one key a transform holds, once its cause is fixed;status/2reports how many it holds (held_keys).status/2also reports a page the drain keeps failing on with nothing charged or paused (drain_failure, aTrellis.DrainFailure), and a finishedself_check/3job's report lists every one open (drain_failures).status/2names the paused upstream columns (paused_upstream_columns, each"target.column") that make a:livetransform report:catching_up.status/2lists the source indexes aGROUP BYorJOINtarget needs and lacks (source_index_gaps, each aTrellis.SourceIndexGapwithtable,columns,role(:group_by,:join_fromor:join_joined: the part of the definition that needs it),opted_outand acreate indexstatement asfix);self_check/3refuses such a target unless it was definedWITHOUT SOURCE INDEX.status/2lists the source indexes aMIN/MAXaggregate's probes read and the source lacks (extreme_index_gaps, each aTrellis.ExtremeIndexGapwithtable,columns,argument,opted_outandfix); nothing pauses, and a deleted extreme is re-read by scanning its group.status/2lists the partitions of a partitioned source a transform is being corrected for while it reads:backfillingfor them (correcting, each aTrellis.CorrectingNodewithnode,sinceandkeys_remaining).status/2names what a Re-derive build waits on instead of its own work (build_wait_xid, the transaction id of the fence it waits out,nilwhen it waits on nothing). While a process of another Trellis version is live,status/2reports that this process stands by (version_handover, aTrellis.VersionHandoverwiththis_version,processes, each aTrellis.OtherProcesswithworker_idandversion(nilfor a release before 0.2.0),unregistered_staging_workerandmigrations_pending), and state-changing calls refuse with aversion_handovererror until it is gone. While this binary is below the catalog floor,status/2reports that this process stands by (below_catalog_floor, aTrellis.BelowCatalogFloorwithfloorandthis_version), and state-changing calls refuse with aversion_floorerror. For aJOINtransformstatus/2reports its hidden tables (join, aTrellis.JoinStatus):hidden_tables, each aTrellis.HiddenTablewithtable,name,estimated_rowsandbytes, theP-size warning (partials_warning, aTrellis.PartialsWarning), a 1-1JOIN's fan-out warnings (fan_out_warnings, oneTrellis.FanOutWarningper joined position past the threshold, naming itsposition) and the step its build runs (building, aTrellis.JoinBuildStepwithstepand theside,part,leaf,remainingorpositionits kind of step carries).joinisnilfor any other transform.status/2reports the oldest Trellis release that can run a transform (floor,"major.minor.patch"), which an application reads before rolling back to an earlier Trellis release (upgrading).request_backfill/2,detect_capture_disruptions/1(pauses now the readers of every table whose Trellis capture was changed outside Trellis, and returns the tables),has_live_drain_workers/1,has_live_staging_worker/1,watermark_token/1,await_converged/3.self_check/3andself_check_job/2: check a target against a fresh recompute from its source, as a background job.self_check/3starts it and returns aTrellis.SelfCheckJob(id,target,mode,state,rows_compared,report,error) at once; a drain worker runs it, andself_check_job/2reads it back byiduntilstateis:done(:queuedand:runningare in progress;:failedand:cancelledcarry anerror), whenreportis aTrellis.SelfCheckReport(outcome,divergences,rows_compared,truncated, ...). A target whose job is still unfinished gets that job back; the next call replaces a finished one, andself_check_job/2returnsnilfor the replaced job or one whose transform was dropped. With no drain worker anywhere in the fleet a job stays:queued.start_link/1, as{Trellis, options}in a supervision tree: a process that owns the handle, so every function above also takes its name.Trellis.Migration:define,applyandstatusin an Ecto migration.schema_dump/1,filter_schema_dump/2,mix trellis.dump,mix trellis.loadandTrellis.StructureDump: the definitions in the structure file, restored when it loads (see Schema dumps).Trellis.Metrics.render_prometheus/0: the process-wide metrics registry as Prometheus text, for a/metricsroute the host already serves (the binding opens no port). It needs no handle. Every series carries thetrellis_instancelabel (<database>/<catalog schema>, as in the log lines), so handles in one VM stay apart.Trellis.LogBridge: started with the:trellis_pgapplication, it forwards the engine's log lines toLogger(domain: [:trellis], the Rust module as:targetmetadata). Every line names its instance in the text, astrellis_instance=<database>/<catalog schema>, so handles in one VM stay apart. Setconfig :trellis_pg, log_level: :infoto choose the most verbose level forwarded (default:Logger.level(), read once at start), orconfig :trellis_pg, log_bridge: falseto forward nothing. Atracingsubscriber installed by another NIF never sees the engine's lines, since each native library links its owntracing; seeTrellis.LogBridge. A line can be dropped if the bridge falls behind, and a warning says how many; logging never blocks or crashes the engine.
# A deploy's migration step: the defaults run nothing in the background.
{:ok, migrator} = Trellis.connect(url: "postgres://localhost/app")
# TLS options go in the URL (docs/embedding.md, "TLS to Postgres"):
# url: "postgres://db.example.com/app?sslmode=verify-full&sslrootcert=/etc/ssl/ca.pem"
:ok = Trellis.migrate(migrator)
:ok = Trellis.shutdown(migrator)
# The app, once migrated. The staging worker reads the catalog as it starts,
# so a `staging: true` handle can't connect before `migrate/1` has run.
{:ok, trellis} = Trellis.connect(url: "postgres://localhost/app", staging: true, drain_threads: 2)
{:ok, _definition} = Trellis.define(trellis, "TRANSFORM widget_prices FROM widgets SELECT price AS price")
{:ok, %Trellis.Status{status: :live}} = Trellis.status(trellis, "widget_prices")
# A column that paused on bad data: find the rows, fix them, resume it.
{:ok, [%Trellis.QuarantineEntry{target: "widget_prices.price"} | _]} = Trellis.quarantined(trellis)
{:ok, %Trellis.SamplePage{samples: rows, next_cursor: cursor}} =
Trellis.sample_quarantined(trellis, "widget_prices.price", limit: 50)
{:ok, {:resumed, ["widget_prices.price"]}} =
Trellis.apply(trellis, "RESUME TRANSFORM widget_prices.price")
:ok = Trellis.shutdown(trellis)
connect/1's defaults (staging: false, drain_threads: 0) run nothing in
the background. For each instance (catalog schema), some connection in the
fleet must set staging: true and some connection must run drain threads, or
no definition ever reaches :live. Any number of connections may set
staging: true: one takes the producer lock and leads, the rest stand by and
take over when it goes away, so a rolling deploy needs no operator step and
the application boots even when the old worker is still running (see
Rolling deploys).
Resources
At the defaults, a handle holds up to 20 Postgres connections (its call pool,
opened as calls need them and closed after 60 seconds idle) and 3 threads. A
worker handle (staging: true, drain_threads: N) also owns a background pool
of up to 2N + 6 connections (2N + 5 without staging, 2 with staging
alone), closed after 60 seconds idle, and two connections that stay open: the
staging session and one LISTEN shared by its drain threads. It holds 6
threads: drain threads are tasks on its runtime, not OS threads. Its memory is
bounded by row counts, not bytes: about N × 100,000 changed rows' images at
its peak. :max_call_pool_size lowers the 20 (the background pool is sized by
the handle's roles, not by an option), and :worker_threads sets the threads.
Every handle (each {Trellis, name: ...} child) costs this again. The full
accounting is in
docs/embedding.md.
Conventions
- Non-bang functions return
{:ok, value}(or:ok) or{:error, %Trellis.Error{code: code, message: message}}; bang variants return the value or raise that error.codeis one of a closed set of atoms, with:unknownfor a code newer than the binding. - No atom is ever built from a string the database returned: every atom in a result comes from a closed set the NIF allocates when it loads.
- Times are
DateTimes. They cross the NIF as epoch microseconds. - A quarantine target is an address string,
"transform"or"transform.column", exactly asquarantined/1reports it. sample_quarantined/3's cursor andwatermark_token/1's token are opaque: pass back what the previous call returned.- A call on a handle holds no scheduler while it waits. The NIF submits the
call on a normal scheduler and returns a reference at once; the engine
sends the result to the calling process, which waits for it in a
receive. The function is synchronous to its caller, a process killed during a call exits at once, and a call stuck on the database holds no scheduler or thread of its own.connect/1andshutdown/1run on dirty IO schedulers, as do the NIFs that touch no handle;Trellis.Metrics.render_prometheus/0runs on a dirty CPU scheduler. - Every call returns within 30 seconds, counted from when it is submitted
(a database that can't answer in time gives
{:error, %Trellis.Error{code: :timeout}}; the server stops what the call started). Calls run in parallel, on one handle and across processes: a call stuck on a lock doesn't delay another process'sstatus/2. A process's own calls stay in order, because it waits for each reply; calls from different processes have no order. shutdown/1doesn't wait for calls in flight: it cancels them, and each of their callers gets a:validationerror.
In a Phoenix app
Let the application's supervisor own the handle. {Trellis, options} starts
a process that connects with connect/1's options as it starts, hands the
handle to the callers that make calls through its name, and shuts the handle
down when the supervisor stops it:
# config/runtime.exs
database_url = System.fetch_env!("DATABASE_URL")
config :my_app, MyApp.Trellis,
name: MyApp.Trellis,
url: database_url,
# Only the fleet's one worker node runs the background work.
staging: System.get_env("TRELLIS_WORKER") == "true",
drain_threads: if(System.get_env("TRELLIS_WORKER") == "true", do: 2, else: 0)
# lib/my_app/application.ex
children = [
MyApp.Repo,
{Trellis, Application.fetch_env!(:my_app, MyApp.Trellis)},
MyAppWeb.Endpoint
]
# Anywhere in the app: the name stands in for the handle.
{:ok, status} = Trellis.status(MyApp.Trellis, "widget_prices")
Each call runs in the process that makes it, in parallel with every other,
so no process queues behind another's call. The caller has to be on the same
node: a handle works only on the node that connected it. shutdown/1 refuses
the name; the supervisor stops it, which cancels the calls still in flight.
Define transforms in Ecto migrations with Trellis.Migration. Trellis never
joins a migration's transaction, so a migration that uses it must set
@disable_ddl_transaction true (the helpers raise before applying anything
if a transaction is open), and must define up/0 and down/0:
defmodule MyApp.Repo.Migrations.DefineWidgetPrices do
use Ecto.Migration
use Trellis.Migration
@disable_ddl_transaction true
def up do
define "TRANSFORM widget_prices FROM widgets SELECT price AS price"
end
def down do
apply "PAUSE TRANSFORM widget_prices"
apply "DROP TRANSFORM widget_prices"
end
end
Ecto doesn't start the application to migrate, so each helper connects a
handle of its own, from the repo's :trellis configuration
(config :my_app, MyApp.Repo, trellis: [url: database_url]), and runs
migrate/1 on it first. A define isn't idempotent, and the helpers don't
make it so: Ecto's schema_migrations runs each migration once. See
Trellis.Migration for the details, and for an optional guard.
Add import_deps: [:trellis_pg] to the app's .formatter.exs to keep
define and apply free of parentheses, like Ecto's own commands.
Fixtures and bulk loads in tests
A loader that disables triggers for the load (ALTER TABLE ... DISABLE TRIGGER ALL, pg_restore --disable-triggers) disables Trellis's capture
triggers too, and ENABLE TRIGGER ALL leaves them in a mode Trellis treats as
a disruption. The rows it wrote never reached a target, so Trellis pauses
every definition that reads the table (capture_failure has kind: :disruption) instead of letting a test read silently wrong data, and waits
for a RESUME. Trellis.TestSupport does that RESUME for a suite, then
waits for the rebuilt targets to catch up:
setup do
# ... load the fixtures with triggers disabled, then re-enabled ...
{:ok, _resumed} = Trellis.TestSupport.resume_capture_disruptions(trellis)
:ok
end
resume_capture_disruptions/2 first pauses the readers of any disrupted
table itself (Trellis.detect_capture_disruptions/1), so it doesn't wait for
the staging worker's next pass; resumes only the definitions paused with kind
:disruption; waits until each is :live and every write committed so far
has reached its targets; and returns {:ok, resumed} ([] when nothing was
disrupted, which costs one catalog read per captured table). A transform an
operator paused, or one paused by a schema change or another failure, stays
paused. A RESUME the engine refuses (a table that is gone, a table define
would now refuse) leaves that definition paused; the rest are still resumed,
and the call then returns {:error, {:resume_refused, refused}}, a map of
target to reason. resume_capture_disruptions!/2 returns the targets and
raises Trellis.TestSupport.ResumeRefused (or Trellis.Error). The
:timeout_ms option (default 30 000) bounds the wait. The handle needs its
staging worker and a drain worker (staging: true, drain_threads: 1),
because the resume rebuilds through them.
Trellis sees a write when it commits, so with Ecto's SQL sandbox, load the
fixtures that targets must reflect outside the sandbox transaction (in
setup_all, or with the sandbox in :auto mode) and call the helper once.
Schema dumps
mix trellis.dump writes the structure file mix ecto.dump writes, with
the definitions in it the way Ecto puts schema_migrations in it, and
mix trellis.load loads it. Make them what mix ecto.dump and
mix ecto.load run, by aliases in mix.exs:
defp aliases do
["ecto.dump": "trellis.dump", "ecto.load": "trellis.load"]
end
The file has the application's tables, the target tables (with their
indexes and grants), the schema_migrations rows, and one row per
definition: its source, target and JOIN table, its text as written, its
grammar version and its floor. Trellis's own schema, the capture triggers on
source tables and the 1-1 targets' ledgers are not in it, so regenerating
the file changes it only where a definition changed. mix trellis.load runs
mix ecto.load and then Trellis.migrate/1 for each repo with a :trellis
configuration, which creates Trellis's tables and restores each definition
into its dumped target. Each is restored unbuilt, and the worker builds it
from the source rows; a paused or quarantined transform is restored
unpaused, because a pause is runtime state. A file loaded another way
(psql -f) needs Trellis.migrate/1 after it. How the restore works, and
what it refuses, is in
docs/embedding.md.
mix ecto.dump can't pass pg_dump extra arguments, so for a repo with a
:trellis configuration (as for Trellis.Migration) mix trellis.dump
runs pg_dump itself, with ecto.dump's arguments (-r, -d, --prefix,
-q) and Trellis.schema_dump/1's, which leave Trellis's schema and the
ledgers out. With --prefix, the instance's target schema is dumped too. It
takes the capture triggers out of the output (Trellis.filter_schema_dump/2),
and reads the definitions before and after the pg_dump, redoing the dump,
up to three times, when a define or drop committed in between, so the
file's DDL and definitions agree. A repo with no :trellis configuration
is dumped by mix ecto.dump.
mix trellis.dump also dumps a repo's :migration_repo, as mix ecto.dump
does, before the Trellis repo's file, and with --prefix it reads
schema_migrations from the schemas named only (not the target schema it adds
to the dump). The repo and its :trellis url have to name the same database,
by host, port and database name spelled alike (localhost is not
127.0.0.1; a repo connecting over a socket directory, or naming no
host, matches a local host), or mix trellis.dump and mix trellis.load
refuse, before anything is written or loaded. A dump that fails once
pg_dump has written the file removes it, since its raw pg_dump output
would load without the definitions; a file no pg_dump wrote to is left as
it was. mix trellis.load restores nothing for a repo mix ecto.load
left alone (--skip-if-loaded on a loaded database, or a "no" to its
prompt).
pg_dump 17 writes CREATE SCHEMA public; when it dumps public by name,
which stops mix ecto.load into a database mix ecto.create made; the task
writes CREATE SCHEMA IF NOT EXISTS public; instead.
A structure file written by an earlier release, which carries the Trellis
schema and the definitions' catalog rows, still loads.
Trellis.StructureDump.append/2 writes that format onto a structure file
someone else dumped with the Trellis schema in it (reading the definitions
once, so it can't see a define racing that dump), and
Trellis.dump_definitions_sql/1 returns its SQL.
Layout
lib/: the publicTrellismodule and its structs, the process that owns a supervised handle (lib/trellis/owner.ex), andTrellis.Migration, which is compiled only when the host depends onecto_sql.lib/trellis/test_support.ex:Trellis.TestSupport, the test helper above.native/trellis_nif/: the NIF crate, a member of the repository's Cargo workspace. The release makes the packaged copy stand alone (see Releasing). Plain-data conversion and error codes come fromclients/embed(trellis-embed), shared with the Ruby binding.
Development
The Erlang and Elixir versions are pinned in .tool-versions. From this
directory:
cargo build -p testkit --bin trellis-testkit # the suite's throwaway Postgres
mix deps.get
mix test
In this repository (:dev and :test) the NIF always builds from source,
never downloads. mix test compiles it through Cargo into the workspace's
target/, and the suite spawns trellis-testkit (from PATH, else the
workspace's debug build) for a Postgres cluster that is torn down when the
test run exits. It needs the Postgres server binaries on PATH, like the
Rust tests.
test/trellis/parity_test.exs runs clients/parity/cases.json, the fixture
the Ruby suite runs too, so the two bindings can't drift apart unnoticed
(see clients/parity/README.md).
Releasing
trellis_pg is released with the rest of Trellis, from one v<version> tag
(docs/releasing.md). @version in mix.exs and
the NIF crate's version (native/trellis_nif/Cargo.toml) are the release's.
.github/workflows/release.yml builds the NIF for each target above and
attaches them to the tag's GitHub release, where lib/trellis/native.ex
downloads them from. The package is then published to Hex by hand with
.github/scripts/publish-packages.sh, which writes
checksum-Elixir.Trellis.Native.exs from the NIFs that release serves.
.github/scripts/build-elixir-nif.sh <target> <out-dir> builds one NIF the
way the workflow does. Linux targets need cargo-zigbuild and Zig
(pip install cargo-zigbuild ziglang).