Concord

A distributed, strongly-consistent embedded key-value store built in Elixir using Viewstamped Replication.

Concord is an embedded database for Elixir applications — think SQLite for distributed coordination. Starts with your application, no separate infrastructure needed. Strong consistency guarantees with ETS-backed read performance.

Key Features

• Strong Consistency — Viewstamped Replication ensures all nodes agree on data
• High Performance — ETS-backed reads with microsecond-level latency
• Embedded Design — Starts with your app, no external infrastructure
• Linearizable Reads — Eventual, leader, and strong names use the same quorum-confirmed VSR barrier
• TTL Support — Automatic key expiration with time-to-live
• Bulk Operations — Efficient batch processing (up to 500 items)
• Value Compression — Automatic compression for large values
• Conditional Updates — Compare-and-swap for optimistic concurrency
• Secondary Indexes — Query by indexed fields
• Backup/Restore — Compressed backups with integrity verification
• Telemetry — Built-in telemetry events for observability hooks

Installation

def deps do
  [{:concord, "~> 3.0.0-beta"}]
end

Quick Start

# Store and retrieve data
Concord.put("user:1001", %{name: "Alice", role: "admin"})
{:ok, user} = Concord.get("user:1001")

# TTL (auto-expires after 1 hour)
Concord.put("session:abc", session_data, ttl: 3600)
{:ok, {data, remaining_ttl}} = Concord.get_with_ttl("session:abc")

# Bulk operations
Concord.put_many([{"k1", "v1"}, {"k2", "v2", 600}])
{:ok, results} = Concord.get_many(["k1", "k2"])

# Conditional update (compare-and-swap)
Concord.put_if("counter", 1, expected: 0)

# All accepted consistency names currently use a linearizable VSR query barrier
Concord.get("key", consistency: :eventual)
Concord.get("key", consistency: :leader)
Concord.get("key", consistency: :strong)

Storage APIs

Concord's default API uses the VSR-backed cluster:

Concord.put("cluster:key", "value")
Concord.Cluster.put("cluster:key", "value")

For data that must stay on only the current node, call the local API:

Concord.Local.put("local:key", "value")
Concord.Local.KV.put("local:record", %{value: 1})

The local API uses the same Concord command/query semantics with separate node-local ETS tables. It does not submit writes to VSR and does not replicate data to cluster peers.

For durable node-local storage backed by Turso/libSQL, enable the Turso engine and call Concord.Turso:

Concord.Turso.put("turso:key", %{value: 1})
Concord.Turso.get("turso:key")
Concord.Turso.txn(%{
  compare: [{:exists, "turso:key", :==, true}],
  success: [{:put, "turso:key", %{value: 2}, %{}}],
  failure: []
})

Concord.Turso uses ex_turso and persists data to a local database file. It does not submit writes to VSR and does not provide Concord cluster membership, leases, watches, or secondary indexes. If remote_url (supporting turso://, libsql://, or https:// schemes) and auth_token are configured, Concord.Turso.sync/1 triggers Turso Cloud sync.

Use key-only pages to enumerate keys without loading stored values:

Concord.Turso.list(prefix: "sessions/", limit: 100, keys_only: true, timeout: 5_000)

These pages preserve record metadata and use covering metadata indexes. Capture {:ok, r} = Concord.Turso.revision() and pass revision: r to each page and Concord.KV.get(key, engine: :turso, revision: r) to read the same retained snapshot despite later appends, overwrites or deletions. Both lexical orders and cursor continuation support that revision. Historical selection seeks bounded batches of matching keys; key-only reads do not fetch stored values. Historical lists return snapshot Records, including their actual metadata. Concord.KV.get/2 returns a historical value even with metadata: true: a non-nil :revision takes precedence. Current point Records are available when :revision is nil or omitted. For snapshot metadata and selected point values, use key-only lists and point reads at the same revision; do not substitute current metadata for historical list Records. TTL visibility still uses the current wall clock. Future revisions are rejected; database resets invalidate old snapshots, and history is not automatically pruned. Older releases discarded prior versions on touch; those missing versions cannot be recovered. Capture revisions after all writers are upgraded for complete snapshots; older reads use the history that was actually retained.

Existing databases backfill current and historical metadata projections during readiness, reading each existing record once; plan an upgrade window for large stores. Original record/history bytes and revision counters remain unchanged. The projections and key-seek indexes add storage and write work proportional to keys, history and metadata. Before serving requests, Concord.Turso.list(prefix: "", limit: 1, keys_only: true, timeout: :infinity) can complete the migration without a finite deadline. By default, Turso list calls return {:error, :timeout} immediately if the connection pool is busy. Use queue: true with a finite positive timeout to wait for brief contention within the same admission, checkout, readiness and query deadline. Queued lists share a finite capacity (default 64), configured by list_queue_capacity; full capacity returns {:error, :overloaded}. Synchronous native work cannot be forcibly interrupted.

Point, metadata and batch reads also preserve one finite timeout through readiness, checkout and SQL. All batch keys share that budget; expiry returns {:error, :timeout} and stops subsequent SQL. Public value and metadata getters check the budget around decompression. Native work or decoding already in progress can finish after expiry, so this is not a hard wall-clock cancellation guarantee. Pass the caller's remaining timeout to each call when composing a read and a later revision CAS; those calls remain separate operations.

Use Concord.Turso.put_many_if/2 to append exact new keys and update related metadata atomically, fenced by the metadata revision you observed:

Concord.Turso.put_many_if([
  {"session/1/turn/002", %{text: "new turn"}},
  {"session/1/meta", %{title: "Updated"}}
], compare: [{:mod_revision, "session/1/meta", :==, observed_revision}])

Only supplied keys change; omitted earlier turns and session values retain their records and history. The operation admits 500 total pairs, including metadata, with 16 MiB logical/encoded values and a 64 MiB logical/encoded command including keys and predicates, subject to lower configured caps. Predicates retain their transaction limits; puts do not consume the generic 128-op/1 MB transaction envelope. A successful nonempty batch shares one new revision and returns {:ok, %{succeeded: true, revision: revision, written: count}}. A conflict returns succeeded: false, the current revision and written: 0 without mutation. Empty batches compare without advancing the revision. Existing keys retain their creation revision and increment their version, as with ordinary pair puts.

Validation, compression, acquisition, readiness and SQL share one :timeout budget (default 5000 ms). Expiry before commit rolls back. An in-flight native commit can finish late, and known successful acknowledgements are preserved. There is no idempotency receipt; reconcile uncertain outcomes before retrying. Use a fixed committed revision across multi-call historical readers.

Use Concord.Turso.replace_prefix/3 to publish a complete transcript and related metadata in one commit:

Concord.Turso.replace_prefix("session/1/turn/", [
  {"session/1/turn/001", %{text: "replacement"}},
  {"session/1/meta", %{title: "Updated"}}
], compare: [{:mod_revision, "session/1/meta", :==, observed_revision}])

The operation removes stale prefix keys, overwrites retained keys, and writes all supplied pairs at one revision. The 501-pair total bound admits 500 replacement rows and one related metadata row. The :max_batch_size setting can lower that bound; an explicit setting of 500 must be raised to 501 for the additional pair. Generic batches stay capped at 500, while 16 MiB values, the 64 MiB command envelope and generic transaction limits retain existing admission checks. Comparisons fence the entire commit; false predicates leave data and revision unchanged. Deletion work depends on selected old keys. Timeout after native commit admission can have an uncertain outcome; reconcile stored state before retrying. Use a fixed revision for multi-call readers to observe a complete old or new publication.

Applications that only need the durable Turso KV engine can disable the Concord VSR cluster runtime while still starting the Turso pool:

config :concord,
  cluster_enabled: false,
  turso: [
    enabled: true,
    database: "./data/turso.db",
    pool_size: 1
  ]

For applications that need a regular Ecto SQL repository backed by Turso/libSQL, use the optional Ecto adapter shipped by ex_turso:

def deps do
  [
    {:concord, "~> 3.0.0-beta"},
    {:ex_turso, "~> 3.0.0-beta"},
    {:ecto_sql, "~> 3.14"}
  ]
end

defmodule MyApp.Repo do
  use Ecto.Repo,
    otp_app: :my_app,
    adapter: Ecto.Adapters.Turso
end

config :my_app, MyApp.Repo,
  database: "my_app.db",
  pool_size: 5

Ecto.Adapters.Turso is the supported equivalent when you need schema/query semantics, migrations, constraints, indexes, transactions, and map/JSON fields. Swap the adapter and connection options in the host application when using PostgreSQL instead.

Multi-Node Cluster

Every replica must use the same ordered membership list. Bootstrap is enabled only for the first start of a new cluster; restart durable replicas with CONCORD_VSR_BOOTSTRAP=false.

export CONCORD_VSR_MEMBERS=n1@127.0.0.1,n2@127.0.0.1,n3@127.0.0.1
export CONCORD_VSR_BOOTSTRAP=true

CONCORD_VSR_REPLICA_ID=n1@127.0.0.1 iex --name n1@127.0.0.1 --cookie concord -S mix
CONCORD_VSR_REPLICA_ID=n2@127.0.0.1 iex --name n2@127.0.0.1 --cookie concord -S mix
CONCORD_VSR_REPLICA_ID=n3@127.0.0.1 iex --name n3@127.0.0.1 --cookie concord -S mix

Performance

Performance varies significantly depending on hardware, cluster size, and network topology. All accepted consistency names use the same VSR read path. Run mix run benchmarks/run_benchmarks.exs on your own hardware to get representative numbers.

When to Use Concord

Comparison

Revision CAS for large values

{:ok, %{succeeded: true, revision: revision, current_kv: %{value: nil}}} =
  Concord.compare_and_swap("history", history, mod_revision: 0)

{:ok, %{succeeded: succeeded, current_kv: current}} =
  Concord.compare_and_swap("history", updated_history, mod_revision: revision)

compare_and_swap/3 atomically compares the visible modification revision and stores one new value, up to the normal 16 MiB logical value limit. Revision zero means absent, deleted or expired. Conflicts return current value-free metadata without advancing the revision. The existing KV.update_if/3 transaction wrapper and generic transactions retain their 1,000,000-byte cap. Concord.Local and Concord.Turso expose the same CAS API; Turso rejects leases.

VSR requires explicit command_version: 2 on every writer after all replicas have been upgraded. The default remains 0; writers 0/1 reject CAS with {:command_version_required, 2}. Follow the reader-first rollout in configuration; v2 WAL/state has no downgrade guarantee. One timeout budget covers validation through commit checks; an already admitted engine write can finish after a caller timeout. A timeout is not proof of no effect; reconcile the stored revision/history before retrying. Known successful native commits retain their success ACK.

Documentation

• Getting Started — Installation, quick start, common use cases
• Elixir API Guide — Read consistency aliases, CAS, queries, compression
• Backup & Restore — Data safety and disaster recovery
• Configuration — All configuration options
• Architecture — Design blueprint

Contributing

1. Fork the repository
2. Create a feature branch
3. Add tests for new functionality
4. Ensure all tests pass (mix test)
5. Submit a pull request

License

MIT License — See LICENSE for details.

Acknowledgments

• The Viewstamped Replication and Viewstamped Replication Revisited papers