ExScylla
An Elixir wrapper around the scylla-rust-driver using Rustler.
- Minimal interface & abstraction
- Easy to maintain & to add new features as they arrive in the rust driver
- Users of ExScylla should be able to leverage the rust driver documentation
Documentation and API examples are available at: https://hexdocs.pm/ex_scylla Technical architecture and design docs can be found in the docs/ folder. Agent instructions and guidelines are located in AGENTS.md.
Prerequisites
- mise: Language runtime manager for Erlang, Elixir, and Rust (
mise install). - Docker: Required for running ScyllaDB test containers and benchmarks.
- scylla_unstable flag: Enabled automatically by
native/ex_scylla/build.rsandnative/ex_scylla/.cargo/config.toml.
Architecture & Performance
NIF/Rustler & Tokio Runtime
ExScylla uses Rustler to interface with the scylla-rust-driver. Because the Rust driver is heavily asynchronous and relies on the Tokio runtime, ExScylla spins up a dedicated Tokio runtime in the background to handle the driver's futures.
Resource Reference Model & Cloning
SessionBuilder, Session, and statements (Batch, Query, Prepared) use Arc references to the underlying Rust objects. These references are passed to Elixir as NIF resources.
Performance Note: Statement setters (Query, Prepared, Batch) return a new reference to an updated copy (cloning the underlying Rust struct). As old instances are not immediately removed (waiting for Elixir GC), these setters should be avoided in the hot path. It is better to create and configure a statement once, then reuse it multiple times. Builders (SessionBuilder, ExecutionProfileBuilder, DefaultPolicyBuilder, LatencyAwarenessBuilder) are different: their setters mutate the builder in place and return the same reference.
Async API & Timeouts
The library uses asynchronous NIFs for non-blocking operations to avoid starving the Erlang schedulers. An async NIF returns immediately and later sends {opaque, result} to the calling process. The async_* functions have no timeout. The synchronous wrappers (for example Session.query/3) wait for the reply with a default timeout of 5 seconds, set by an optional trailing timeout_ms argument.
Many requests from one process: replies are sent from several Tokio threads at once. A process that keeps many async_* requests in flight should call Process.flag(:message_queue_data, :off_heap), which avoids mailbox lock contention (about 15–20% faster for 1000 concurrent inserts in our benchmarks).
Dual API (Decoded vs Raw)
Both APIs call the same NIFs. Rust always returns the result rows as a raw binary, and the two APIs differ only in how Elixir decodes it:
- Decoded API (
query,execute,batch): returns%ExScylla.Types.QueryResult{}whose rows are%ExScylla.Types.Row{columns: [...]}structs of type-tagged values such as{:int, 42}. - Raw API (
query_raw,execute_raw,batch_raw): returns%ExScylla.Types.QueryResultRaw{}whose rows are plain lists of untagged values. This skips building the type tags.
Example
This example assumes a running ScyllaDB node and an existing test.s_doc table with columns a text, b int, c double.
alias ExScylla.SessionBuilder
alias ExScylla.Session
alias ExScylla.Types.QueryResult
{:ok, session} = SessionBuilder.new()
|> SessionBuilder.known_node("127.0.0.1:9042")
|> SessionBuilder.build()
{:ok, ps} = Session.prepare(session, "INSERT INTO test.s_doc (a, b, c) VALUES (?, ?, ?)")
values = [{:text, "test"}, {:int, 2}, {:double, 1.0}]
{:ok, %QueryResult{}} = Session.execute(session, ps, values)
# Iterative queries (streams)
Session.query_stream(session, "SELECT * FROM test.s_doc", [])
|> Enum.each(fn row -> IO.inspect(row) end)
Installation
https://hex.pm/packages/ex_scylla
def deps do
[
{:ex_scylla, "~> 0.11.0"}
]
end
Repository Blueprint
System Overview
ExScylla has two layers that must stay in sync:
- Elixir API (
lib/ex_scylla):SessionBuilder,Session, statements, execution profiles, load-balancing builders, result types, andCqlTypesrow decoding. Wrappers are generated byExScylla.Macros.Nativeand call stubs inExScylla.Native. - Rust NIF crate (
native/ex_scylla): a Rustlercdylibloaded asElixir.ExScylla.Native. It wraps thescylla1.10.0 driver objects asResourceArcresources and runs all network I/O on one shared Tokio runtime.
Request flow for an I/O call such as Session.query/3:
- The Elixir wrapper calls
ExScylla.Native.s_query(opaque, ...). The defaultopaqueis{function_name, make_ref()}. - The NIF decodes the arguments synchronously, spawns a future through
async_elixir!, and returns:okright away. - When the future finishes, the caller receives
{opaque, result}. The synchronous wrapper waits up totimeout_ms(default 5000), then returns{:error, :timeout}. - Result rows reach Elixir as a raw, resource-backed binary (
ScyllaRawRowsResource).ExScylla.Types.QueryResult.decode/1(decoded API) ordecode_raw/1(raw API) parses it in Elixir. Both APIs use the same NIFs.
Local Interface & Invariants
mix.exs(ExScylla.MixProject): app:ex_scylla, version0.11.0.elixirc_paths/1addstest/utilsfor:testand:bench.- Aliases:
testre-runsmix test.wrapper(which callsMix.Tasks.Test.run/1) in a subprocess. It passesMIX_ENV,RUSTFLAGS="-C instrument-coverage --cfg scylla_unstable"and--color/--no-color.bench(run_bench/1) runsmix run bench/<name>.exswithMIX_ENV=bench.
- The Hex package (
package/0) shipslib,priv/native/.place_holder,native/ex_scylla/src,native/ex_scylla/Cargo.*,native/ex_scylla/build.rs,native/ex_scylla/.cargo/config.toml(needed for--cfg scylla_unstable),.formatter.exs,mix.exsandREADME*.
.formatter.exs: formats{mix,.formatter}.exsand{config,lib,test}/**/*.{ex,exs}.mise.tomldefines all workflow tasks (deps:get,compile,lint,format,build,test,test:coverage,install-test-deps,bench,scylla:*). Add new commands there, not in aMakefile.- The package version in
mix.exsand the crate version innative/ex_scylla/Cargo.tomlmove together (both0.11.0).
Subdirectories & System Boundaries
- lib: ./lib/README.md - Elixir source root. ./lib/ex_scylla/README.md covers
ExScylla.Nativestubs,SessionBuilder,Session(decoded, raw and stream APIs) andCqlTypes. It has children for execution, load_balancing, macros, statement and types (with errors). - native: ./native/README.md - the
ex_scyllaRust crate (./native/ex_scylla/README.md) and its sources (./native/ex_scylla/src/README.md). Sources hold NIF handlers per area plus resource and type modules in batch, execution, prepared_statement, query, session and session_builder. - bench: ./bench/README.md - standalone Benchee scripts comparing ExScylla with
erlcassagainst a Testcontainers ScyllaDB instance. Run withmise run bench <name>. - config: ./config/README.md - Mix config entry point that imports
<env>.exs. Sets no application keys. - Not covered by contracts:
test/(ExUnit suite andtest/utils/test_support.ex),docs/(prose design docs, see docs/README.md), andpriv/(the compiled NIF library plus the trackedpriv/native/.place_holder, which ships in the Hex package).
Global Invariants
- Dual-layer sync. Every public function needs a Rust NIF in
native/ex_scylla/src/, a stub with the same arity inlib/ex_scylla/native.ex, and a wrapper inlib/ex_scylla/. EveryNifStruct#[module = "ExScylla.Types.*"]and its fields must match the Elixir struct. - No blocking in schedulers. Network I/O goes through
async_elixir!and replies with exactly one{opaque, result}message. Never useblock_on. - Copy-on-write statements, in-place builders.
Query,PreparedandBatchsetters clone and return a new reference.SessionBuilder,ExecutionProfileBuilder,DefaultPolicyBuilderandLatencyAwarenessBuildersetters mutate under aMutexand return the same reference. - Rows are decoded in Elixir. Rust never builds row terms.
ScyllaValue(Rust, used for bound values) andExScylla.CqlTypes.decode_value/2(Elixir, used for results) must agree on the CQL type mapping. - Unstable driver APIs. The crate must build with
--cfg scylla_unstable, whichnative/ex_scylla/build.rsandnative/ex_scylla/.cargo/config.tomlboth set.
Global Side Effects
- Loading the NIF starts a process-global multi-threaded Tokio runtime that lives as long as the VM.
SessionBuilder.buildopens TCP connections to ScyllaDB/Cassandra nodes, and the driver runs background metadata, keepalive and pool tasks.- Async results are delivered to the calling process mailbox. A reply that arrives after a sync timeout stays in the mailbox.
- Tests and benchmarks start a
scylladb/scylla:2026.3Docker container through Testcontainers. Test builds write LLVM coverage profiles (*.profraw).
System Verification
- Run:
mise run test(ExUnit suite; requires Docker) - Run:
mise run lint(format check, warnings as errors,cargo check, clippy)
Contributing
We welcome contributions! Please see the CONTRIBUTING.md file for guidelines on how to contribute to this project.
Changelog
For a detailed list of changes and updates, please refer to the CHANGELOG.md file.