barrel_docdb

Alpha Software - API may change. Feedback welcome via GitHub Issues.

Overview

barrel_docdb provides:

• Document CRUD with MVCC revision trees and conflict resolution
• Declarative Queries with automatic path indexing
• Real-time Subscriptions via MQTT-style path patterns and queries
• Changes Feed with long-poll and Server-Sent Events
• Attachments with streaming for large binaries
• HTTP API with REST endpoints for all operations
• Peer-to-Peer Replication (one-shot or continuous) with pluggable transports
• Prometheus Metrics for monitoring and alerting
• HLC Ordering for distributed event coordination

Use Cases

• Edge Computing: Deploy nodes that sync to cloud when connected
• Multi-Region: Replicate data across regions with conflict resolution
• Offline-First Apps: Full MVCC for seamless sync
• Application Backend: Document storage with queries, changes feed, and attachments

Quick Start

%% Start the application
application:ensure_all_started(barrel_docdb).

%% Create a database
{ok, _} = barrel_docdb:create_db(<<"mydb">>).

%% Save a document
{ok, #{<<"id">> := DocId, <<"rev">> := Rev}} = barrel_docdb:put_doc(<<"mydb">>, #{
    <<"type">> => <<"user">>,
    <<"name">> => <<"Alice">>
}).

%% Fetch the document
{ok, Doc} = barrel_docdb:get_doc(<<"mydb">>, DocId).

%% Query documents
{ok, Users, _Meta} = barrel_docdb:find(<<"mydb">>, #{
    where => [{path, [<<"type">>], <<"user">>}]
}).

%% Subscribe to changes (MQTT-style patterns)
{ok, SubRef} = barrel_docdb:subscribe(<<"mydb">>, <<"type/user/#">>),
receive {barrel_change, _, Change} -> io:format("~p~n", [Change]) end.

HTTP API

The HTTP server starts automatically with the application when {http_enabled, true} is set (the default). Port defaults to 8080; override via the http_port config key or the BARREL_HTTP_PORT environment variable. See Configuration below.

Basic Operations

# Health check
curl http://localhost:8080/health

# Create database
curl -X PUT http://localhost:8080/db/mydb

# Create document with auto-generated ID
curl -X POST http://localhost:8080/db/mydb \
  -H "Content-Type: application/json" \
  -d '{"type": "user", "name": "Alice"}'

# Or create document with specific ID
curl -X PUT http://localhost:8080/db/mydb/doc1 \
  -H "Content-Type: application/json" \
  -d '{"type": "user", "name": "Alice"}'

# Get document
curl http://localhost:8080/db/mydb/doc1

# Query documents
curl -X POST http://localhost:8080/db/mydb/_find \
  -H "Content-Type: application/json" \
  -d '{"where": [{"path": ["type"], "value": "user"}]}'

Changes Feed

# Poll for changes
curl "http://localhost:8080/db/mydb/_changes?since=first"

# Long-poll (wait for changes)
curl "http://localhost:8080/db/mydb/_changes?feed=longpoll&timeout=30000"

# Server-Sent Events stream
curl http://localhost:8080/db/mydb/_changes/stream

Attachments

# Put attachment
curl -X PUT http://localhost:8080/db/mydb/doc1/_attachments/photo.jpg \
  -H "Content-Type: image/jpeg" \
  --data-binary @photo.jpg

# Get attachment
curl http://localhost:8080/db/mydb/doc1/_attachments/photo.jpg > photo.jpg

Replication

Basic Replication

%% One-shot replication
{ok, Result} = barrel_rep:replicate(<<"source">>, <<"target">>).

%% With filter (path patterns)
{ok, _} = barrel_rep:replicate(<<"source">>, <<"target">>, #{
    filter => #{paths => [<<"users/#">>]}
}).

%% With query filter
{ok, _} = barrel_rep:replicate(<<"source">>, <<"target">>, #{
    filter => #{query => #{where => [{path, [<<"status">>], <<"active">>}]}}
}).

HTTP Replication

%% Replicate to remote node
{ok, _} = barrel_rep:replicate(<<"mydb">>, <<"http://remote:8080/db/mydb">>, #{
    source_transport => barrel_rep_transport_local,
    target_transport => barrel_rep_transport_http
}).

Conflict Resolution

barrel_docdb uses revision trees (CRDT-style) for conflict handling:

%% Detect conflicts
case barrel_docdb:get_conflicts(<<"mydb">>, DocId) of
    {ok, []} ->
        %% No conflicts
        ok;
    {ok, Conflicts} ->
        %% Resolve by choosing a winner
        barrel_docdb:resolve_conflict(<<"mydb">>, DocId, WinningRev, choose)
end.

Prometheus Metrics

Metrics are exposed at /metrics in Prometheus text format:

curl http://localhost:8080/metrics

Available metrics:

Configuration

In your sys.config:

{barrel_docdb, [
    {data_dir, "data/barrel_docdb"},
    {http_port, 8080},
    {http_enabled, true}
]}.

Requirements

• Erlang/OTP 28 or 29
• RocksDB (via rocksdb hex package)

Installation

Add to your rebar.config:

{deps, [
    {barrel_docdb, "0.7.7"}
]}.

Architecture

barrel_docdb_sup
├── barrel_metrics         (OpenTelemetry metrics)
├── barrel_cache           (RocksDB block cache)
├── barrel_hlc_clock       (Hybrid Logical Clock)
├── barrel_sub             (Path subscriptions)
├── barrel_query_sub       (Query subscriptions)
├── barrel_path_dict       (Path interning for posting lists)
├── barrel_query_cursor    (Chunked query cursors)
├── barrel_parallel        (Worker pool for parallel queries)
├── barrel_db_sup          (Database supervisor)
│   └── barrel_db_server   (Per-database process)
├── barrel_rep_tasks       (Replication task manager)
├── barrel_http_api_keys   (HTTP API key management)
├── barrel_peer_auth       (HTTP replication auth)
└── barrel_http_server     (HTTP/2 server, when http_enabled=true)

API Reference

Document Operations

Replication

Attachments & Changes

Support

License

Apache License 2.0. See LICENSE for details.

Built by Enki Multimedia | barrel-db.eu