Sediment

SQLite-compatible database for Elixir on the Turso engine, with optional S3-backed durability.

Status: experimental. Expect rough edges; see Naming and the disclaimer below.

Sediment is an Elixir driver for Turso, the SQLite-compatible database engine written in Rust.

sediment is to Turso what exqlite is to SQLite: a NIF (built with Rustler over turso_core 0.8.1) plus a DBConnection implementation. The public API mirrors exqlite's module by module, so code written against exqlite ports over by renaming modules:

It also exposes Turso-only features: MVCC with BEGIN CONCURRENT, page encryption, vector search, full-text search, change data capture and S3-backed durability.

If you are looking for the Ecto adapter, see ecto_sediment, the Sediment equivalent of ecto_sqlite3.

New to it? Start with the getting started guide.

Naming

You might expect a library like this to be called turso_ex or ecto_turso. Elixir integrations are often named after what they wrap, and that makes them easy to find. We deliberately didn't do that.

We love Turso. The Turso database engine is the foundation of this library. It's an impressive, open-source (MIT) reimplementation of SQLite in Rust, and everything Sediment does stands on that work. But "Turso" is the name of a company and its products, and we don't want to ride on the coattails of their good name:

• This is not an official Turso project, and we don't want anyone to assume it is. Sediment is still experimental. Bugs, rough edges and integration mistakes here are ours, not Turso's, and they shouldn't reflect on the quality of the Turso engine. Please report them to us, not to Turso.
• Sediment takes a different path from Turso's own offering. It uses the open-source engine and adds its own durability layer that stores your database in an S3-compatible bucket you control. Turso Cloud, the company's hosted database, doesn't work with this library right now. Naming the library after Turso would suggest otherwise.

So the name is our own. Sediment describes how the S3 layer works: committed changes settle into the bucket in layers, as log segments that get compacted into snapshots, and together they make up the solid ground your data rests on.

"Turso" appears in these docs only to describe what Sediment is built on: the engine, its compatibility and its behaviour. See also the disclaimer below: Sediment isn't affiliated with Turso.

Not affiliated with Turso. Sediment is an independent, community-maintained Elixir library built on the open-source Turso database engine (turso_core, MIT). "Turso" is a trademark of its respective owner and is used here only to describe compatibility. This project is not endorsed by, sponsored by, or otherwise connected with Turso or its company.

Provided under the MIT License, without warranty of any kind. The underlying engine is pre-1.0; keep independent backups of any data you care about.

Installation

defp deps do
  [
    {:sediment, "~> 0.1"}
  ]
end

With Ecto, depend on the adapter instead; it brings sediment with it:

{:ecto_sediment, "~> 0.1"}

Elixir 1.18 or later; the package is tested on Elixir 1.18 with OTP 27 and Elixir 1.20 with OTP 29, on Linux.

The NIF comes precompiled (via RustlerPrecompiled) for these targets, so no Rust toolchain is needed:

The download is checked against the checksums in the Hex package. glibc 2.28 covers Debian 10+, Ubuntu 20.04+, RHEL/Rocky/Alma 8+ and Amazon Linux 2023, but not Amazon Linux 2 (glibc 2.26). On Alpine the NIF links libgcc_s: install libgcc if your image doesn't have it. The test suites run on Linux only; the other targets are built in the release, not tested.

To build from source instead (any other target, or by policy), set SEDIMENT_BUILD=1 when compiling and install a Rust toolchain (1.91 or later). A git or path dependency on sediment always builds from source: the precompiled NIFs are only used from the Hex package, which carries their checksums.

Caveats

• Sediment.Engine does not cache prepared statements; Sediment.Connection keeps each connection's 64 most recently used ones.
• Prepared statements are not immutable. Do not manipulate a statement concurrently; keep it isolated to one process.
• Native calls that touch the database run on dirty IO schedulers. interrupt/1 and cancel/1 only set a flag and run on normal schedulers, so they are never queued behind busy dirty schedulers.
• Datetimes are stored without offsets, as ISO 8601 text, like exqlite.
• When storing BLOB values, use {:blob, the_binary}, otherwise the value is stored as text. Binaries that are not valid UTF-8 are stored as blobs, because Turso text must be UTF-8.

Configuration

Runtime configuration

config :sediment,
  default_chunk_size: 100,
  type_extensions: [MyApp.TypeExtension]

• default_chunk_size - The chunk size used when multi-stepping without an explicit chunk size.
• type_extensions - An optional list of modules that implement the Sediment.TypeExtension behaviour.

Compile-time configuration

Full-text search pulls in tantivy and adds a few minutes to the first build. It is on by default; to build without it:

config :sediment, Sediment.Native, default_features: false

Usage

The Sediment.Engine module works like Exqlite.Sqlite3:

# We'll just keep it in memory right now
{:ok, conn} = Sediment.Engine.open(":memory:")

# Create the table
:ok = Sediment.Engine.execute(conn, "create table test (id integer primary key, stuff text)")

# Prepare a statement
{:ok, statement} = Sediment.Engine.prepare(conn, "insert into test (stuff) values (?1)")
:ok = Sediment.Engine.bind(statement, ["Hello world"])

# Step is used to run statements
:done = Sediment.Engine.step(conn, statement)

# Prepare a select statement
{:ok, statement} = Sediment.Engine.prepare(conn, "select id, stuff from test")

# Get the results
{:row, [1, "Hello world"]} = Sediment.Engine.step(conn, statement)

# No more results
:done = Sediment.Engine.step(conn, statement)

# Release the statement.
:ok = Sediment.Engine.release(conn, statement)

With DBConnection (pooling, transactions, streams):

{:ok, conn} = Sediment.start_link(database: "app.db", journal_mode: :wal)

Sediment.query!(conn, "create table users (id integer primary key, name text)")
Sediment.query!(conn, "insert into users (name) values (?)", ["Alice"])

{:ok, %Sediment.Result{rows: [[1, "Alice"]]}} =
  Sediment.query(conn, "select id, name from users")

Sediment.transaction(conn, fn conn ->
  Sediment.query!(conn, "update users set name = ? where id = ?", ["Bob", 1])
end)

See Sediment.Connection.connect/1 for all connection options.

Turso features

See the Turso extensions guide for details and examples.

• MVCC and BEGIN CONCURRENT - journal_mode: :mvcc lets several connections write in concurrent transactions; default_transaction_mode: :concurrent makes transactions use BEGIN CONCURRENT. Choose it when creating the database: an existing database with AUTOINCREMENT tables refuses to switch (see Differences from exqlite).
• Encryption - encryption: [cipher: "aegis256", key: hex_key].
• Vector search - built-in vector functions, and Sediment.Vector to encode Elixir lists as vector blobs.
• Full-text search - CREATE INDEX ... USING fts with experimental: [:index_method], queried with fts_match and fts_score.
• Change data capture - Sediment.CDC reads the changes Turso records per connection.
• S3 durability - the database's state of record lives in S3: commits are uploaded in the background (async by default, sync: true or durability: :sync to wait), and a single writer is fenced by the store's conditional writes; also read-only replicas, point-in-time restore and encryption at rest. See the S3 durability guide; the protocol is model-checked with TLA+ (formal model). S3 databases are encrypted by default (:encryption with a key, or encryption: false to opt out explicitly). Opening with s3: restores the S3 copy over the local file; an empty prefix refuses to start over an existing local database, which Sediment.S3.import/3 uploads instead.
• Experimental flags - experimental: [:attach, :views, ...].

Telemetry

Sediment.Telemetry lists the events: spans around preparing and executing statements ([:sediment, :prepare | :query, ...]), pooled connection disconnects with their reason (S3 fencing, a contained turso_core panic, a closed handle), and spans for the S3 operations. For connect and disconnect counts, start the pool with a DBConnection.TelemetryListener in :connection_listeners.

Differences from exqlite

API parity

Every public function of exqlite, per module ("unsupported" functions exist and return {:error, :not_supported}; "no" means the function isn't provided; Exqlite.Sqlite3NIF and Exqlite.Flags are internal modules whose design differs). The exqlite API parity guide lists each function with the reason for every difference.

Turso is SQLite compatible at the file format and SQL level, but it is a different engine. Where Turso cannot behave like SQLite, sediment says so instead of faking it.

Not supported

Behaves differently

• MVCC checkpoint threshold. Non-S3 databases in MVCC mode checkpoint automatically every 256 KiB of logical log (:mvcc_checkpoint_threshold; nil for turso_core's ~4 MB default). The default protects AUTOINCREMENT tables (Ecto's default primary keys), whose inserts slow down as the log grows: with a 1 MiB threshold they commit 8x slower than with 256 KiB. Tables without AUTOINCREMENT go the other way, about 2x faster with 4 MiB, at the price of longer checkpoint pauses; see bench/RESULTS.md.
• Hot rows in long MVCC transactions. In MVCC mode, turso_core 0.8.1 keeps one row version per update, even for updates by the same transaction, and every access walks that chain. So a transaction that updates one row many times gets slower with each update. An AUTOINCREMENT table does that on every insert (its sqlite_sequence row): 500 inserts took 49 ms, and the sixth 500 in the same transaction 252 ms, against a flat 5 ms without AUTOINCREMENT. Prefer INTEGER PRIMARY KEY without AUTOINCREMENT for MVCC tables (Ecto: migration_primary_key: [type: :integer] in ecto_sediment), or keep such transactions short. WAL mode is not affected. With several connections inserting into the same AUTOINCREMENT table in BEGIN CONCURRENT transactions, the id allocations also conflict with each other: with 4 writers, about 9% of transactions failed with "Write-write conflict" (none without AUTOINCREMENT) and throughput was about 5 times lower.
• Journal modes. Turso stores data in WAL mode or MVCC mode. journal_mode: :delete | :truncate | :persist | :memory | :off are accepted and have no effect. :mvcc is Turso only.
• Switching an existing database to MVCC. turso_core 0.8.1 doesn't carry AUTOINCREMENT sequences over when an existing WAL database switches to MVCC: the next insert reuses id 1 and silently overwrites that row. So opening such a database with journal_mode: :mvcc, or running PRAGMA journal_mode = 'mvcc' on it, fails with "refusing to switch to MVCC: ..." and leaves it unchanged. Databases without AUTOINCREMENT tables switch normally, and databases created in MVCC mode (every S3 database) are not affected. To move an existing database with AUTOINCREMENT tables to MVCC, copy its data into a new database opened with journal_mode: :mvcc from the start. The check runs right before the switch (again after any wait for a lock), but the two aren't atomic: don't create AUTOINCREMENT tables from other connections while a database is being switched.
• Pragmas. :case_sensitive_like, :secure_delete, :wal_auto_check_point, :journal_size_limit, :soft_heap_limit and :hard_heap_limit are accepted for compatibility and have no effect. locking_mode: :normal (the default) is not applied, because Turso always holds the database file exclusively per process. auto_vacuum: :full | :incremental requires experimental: [:autovacuum].
• :progress_handler_steps / set_progress_handler_steps/2 are no-ops: interrupt/1 and cancel/1 reach Turso's VM directly.
• Read-only connections. Turso shares one database instance per file in the VM. A read-only open of a file that is already open read-write is enforced per connection (query_only); writes fail with "attempt to write a readonly database" as in SQLite.
• close/1 finalizes the connection's prepared statements, so it releases the database, its file locks and an S3 lease right away even if statement references are still alive (for example cached by Ecto in client processes). Using such a statement afterwards returns {:error, :connection_closed} or {:error, :invalid_statement} (exqlite may still step it).
• Busy waits (a writer waiting for the lock, up to the busy timeout) sleep in the calling process, not on a dirty scheduler thread as SQLite's busy handler does under exqlite: many waiting writers can't take every dirty IO thread and starve the lock holder's commit or other file I/O in the VM. cancel/1 still ends a wait.
• serialize/2 is implemented with VACUUM INTO, so the image is a compacted copy. deserialize/3 only replaces "main"; statements prepared before it must be prepared again.
• Error messages follow SQLite's wording where Turso's is only cosmetically different ("no such table: t", "database is locked"), but syntax errors read differently: "unexpected token: a" instead of near "a": syntax error.
• Constant-folded parameters. turso_core 0.8.1 drops parameters that constant folding removed at the end of a statement (WHERE 0 AND x = ?) from the parameter count. bind/2 accepts arguments for them when the SQL text has exactly that many placeholders, so queries such as Ecto's where: p.id in ^[] work. Named parameters are not covered by this.
• Prepared PRAGMA statements are not re-prepared after schema changes in turso_core 0.8.1: PRAGMA table_info(t) prepared before DROP TABLE t keeps returning the old columns. Prepare such statements again after DDL. Sediment.Connection's statement cache never caches PRAGMAs.
• Schema changes under a write transaction (MVCC). Like SQLite, turso re-prepares a statement whose schema changed, but inside a write transaction it gives up after two attempts: a transaction that overlaps another connection's DDL can fail with "Database schema changed". Roll back and retry the transaction, or run DDL, such as migrations, while nothing else writes.
• FULL OUTER JOIN on an indexed column fails in turso_core 0.8.1 with FULL OUTER JOIN requires an equality condition in the ON clause when the right-hand table's join column has an index (a primary key, say). A unary plus avoids the index: ON +p.id = c.parent_id.
• Dropping a column that has its own REFERENCES. turso_core 0.8.1 refuses ALTER TABLE child DROP COLUMN parent_id when parent_id was declared with a column-level REFERENCES (unknown column "parent_id" in foreign key definition); SQLite drops the constraint with the column. This includes rolling back Ecto's add :parent_id, references(:parents). Rebuild the table instead: create it without the column, copy the rows, drop the old table, rename the new one and recreate its indexes. A table-level FOREIGN KEY (parent_id) is refused by both engines.
• Recursive triggers. PRAGMA recursive_triggers = on is accepted, but turso_core 0.8.1 fires a trigger once instead of recursing.
• Infinite floats (SELECT 1e999) are returned as :inf and :"-inf", like Postgrex; exqlite raises an ArgumentError for them.
• file: URIs are translated to a path and mode (mode=ro, rw, rwc, memory); other URI parameters are ignored.

Performance and testing

bench/ compares sediment with exqlite; see bench/RESULTS.md. Through a DBConnection pool (and so Ecto) queries and single-row inserts are faster than exqlite's; through the low-level API reads are faster and writes 1.2-2x slower (turso_core's insert path plus two native calls per statement); concurrent writers in MVCC mode commit about twice as many transactions per second.

mix test runs the unit and integration suites. Tagged suites: --only s3 (needs an S3 server such as SeaweedFS on 127.0.0.1:8333), --only slow_test (cancellation and a longer fuzz run; the stress, timeout, model and fuzz tests also run in the default suite), --only soak (a mixed five-minute workload, SOAK_SECONDS to change it, that checks memory, NIF resources and scheduler latency stay bounded), and --only torture (S3 crash torture, 30 minutes, TORTURE_MINUTES to change it; see the S3 guide). .github/workflows/ci.yml runs mix ci, --only s3 and --only slow_test on the oldest and newest supported Elixir/OTP versions, plus cargo fmt, cargo clippy and cargo test, against a SeaweedFS service container; the soak and torture suites run locally (they take minutes to half an hour).

Under the hood

Each Sediment.Engine.open/2 creates a Turso connection guarded by a mutex that is held for the duration of every call, like exqlite's per-connection mutex. Turso's statement stepping is asynchronous; the NIF drives Turso's IO loop until a row or completion is available, on a dirty IO scheduler. Busy handler backoff is waited out in short slices so cancel/1 can abort a statement that is waiting for a lock.

turso_core compiles expressions and runs trigger programs recursively, which can need more stack than a dirty scheduler thread has (about 320 KB): a 50-term expression or a chain of 100 triggers would overflow it and crash the VM. So every call into turso_core runs on a 16 MB stack that each dirty IO scheduler thread maps once, with a guard page, and reuses. It is committed lazily, so it costs virtual address space rather than memory.

Acknowledgements

• Built on the open-source Turso database engine (turso_core, MIT).
• Derived from exqlite by Matthew A. Johnston: the API, documentation and many tests are ported from it.
• The NIF is built with Rustler.

License

MIT, see LICENSE. It keeps exqlite's copyright notice for the ported code.

Cargo.lock lists webpki-root-certs (CDLA-Permissive-2.0), which may trip license allow-lists that only cover code licenses. It is a wasm32-only dependency of rustls-platform-verifier and is not built into the NIF on any supported target.