barrel

The embeddable edge-AI database. barrel composes the document layer (barrel_docdb) and the vector layer (barrel_vectordb) behind one API, so an Erlang application can embed a single database that does documents, vectors, BM25, hybrid search, attachments (blobs), and a changes feed.

Documentation | HexDocs | Repository

A barrel database is a docdb database plus a vectordb store that share a name and a single id space: a document, its attachments (blobs), and its vector are all addressed by the same id. Blobs are docdb attachments; the storage backend is pluggable per database via the docdb barrel_att_backend seam (RocksDB BlobDB by default). barrel adds no storage of its own; it coordinates the layers. Each underlying app stays usable on its own.

Open and documents

{ok, Db} = barrel:open(mydb),

{ok, _}  = barrel:put_doc(Db, #{<<"id">> => <<"a">>, <<"title">> => <<"hello">>}),

{ok, Doc} = barrel:get_doc(Db, <<"a">>),

{ok, Rows, _Meta} = barrel:find(Db, #{where => [{path, [<<"title">>], <<"hello">>}]}),

ok = barrel:close(Db).

barrel:open/2 accepts #{docdb => Map, vectordb => Map} to pass options to each layer, including docdb => #{att_opts => #{backend => ...}} to choose an attachment backend.

Batches

[{ok, _}, {ok, _}] = barrel:put_docs(Db, [#{<<"id">> => <<"a">>}, #{<<"id">> => <<"b">>}]),

[{ok, _}, {ok, _}] = barrel:get_docs(Db, [<<"a">>, <<"b">>]),

{ok, #{inserted := 2}} = barrel:vector_add_batch(Db, [

    {<<"a">>, <<"t1">>, #{}, V1},

    {<<"b">>, <<"t2">>, #{}, V2}

]).

vector_add_batch/2 takes {Id, Text, Metadata} (text embedded by the store) or {Id, Text, Metadata, Vector} (explicit) tuples; a batch must be all one shape.

Vectors and search

ok = barrel:vector_add(Db, <<"a">>, <<"hello world">>, #{}, [0.1, 0.2, 0.3]),

{ok, Hits} = barrel:search_vector(Db, [0.1, 0.2, 0.3], #{k => 5}),

{ok, Hits2} = barrel:search_hybrid(Db, <<"hello">>, #{k => 5}).

Attachments

{ok, _} = barrel:put_attachment(Db, <<"a">>, <<"file.txt">>, <<"bytes">>),

{ok, <<"bytes">>} = barrel:get_attachment(Db, <<"a">>, <<"file.txt">>),

[<<"file.txt">>] = barrel:list_attachments(Db, <<"a">>).

Large attachments stream: open_attachment_writer/4 + write_attachment/2 + finish_attachment/1, and open_attachment_reader/3 + read_attachment/1.

Changes

{ok, Changes, LastHlc} = barrel:changes(Db, first),

{ok, StreamPid} = barrel:subscribe(Db, LastHlc).

Identify the embedder

{ok, #{fingerprint := Fp, model := Model, dimensions := Dim}} = barrel:embedder_info(Db).

embedder_info/1 returns provider, model, revision, dimensions, distance, preprocessing and a fingerprint (sha256: of their canonical JSON). Two databases with the same fingerprint produce comparable vector scores. info/1 carries the same identity under embedder. There is no fingerprint without a configured embedder.

Know which state answered a query

query/2,3 and query_fold/5 results carry the database's instance_id and last_seq in their meta, for collection queries and table functions alike:

{ok, _Rows, #{instance_id := Id, last_seq := Seq}} =

    barrel:query(Db, <<"SELECT id FROM c LIMIT 10">>).

Open a copy read only

{ok, Ro} = barrel:open(<<"snapshot">>, #{read_only => true}).

{error, read_only} = barrel:put_doc(Ro, #{<<"id">> => <<"x">>}).

Both stores open read only and write no file; record mode persists no policy and starts no indexer. Add embedding => stored to run record mode with the policy the database persisted ({error, no_stored_policy} on a plain database). A store an older version wrote fails with read_only_upgrade_needed until one writable open upgrades it.

Keep databases open with barrel_dbs

barrel_dbs owns long-lived handles for servers: it opens lazily, closes idle databases and evicts at dbs_max_open.

{ok, Db} = barrel_dbs:ensure(<<"docs">>, #{must_exist => true}),

{ok, Db, Lease} = barrel_dbs:lease(<<"docs">>, #{}),

%% ... the database stays open while the lease is held ...

ok = barrel_dbs:release(Lease).

Export and import a database

Export copies a closed database with a checksummed manifest; import verifies every file and serves the copy read only.

{ok, _} = barrel_ctx_export:export(<<"docs">>, "/srv/export/docs_g1",

                                   #{generation => 1}),

{ok, #{name := Name, db := Copy}} = barrel_ctx_export:import("/srv/export/docs_g1", #{}).

Contexts

barrel_ctx queries several databases, local, imported or on other barrel_server nodes, with one BQL statement, and keeps working sets you can query offline:

{ok, #{<<"id">> := _}} = barrel_ctx:register(

    #{<<"name">> => <<"otp/sasl">>,

      <<"locations">> => [#{<<"kind">> => <<"local">>, <<"db">> => <<"otp_sasl">>}]}),

{ok, #{execution := succeeded, rows := Rows, summary := Summary}} =

    barrel_ctx:query(#{query => <<"SELECT id, lines FROM c ORDER BY lines DESC LIMIT 10">>,

                       contexts => [<<"otp/sasl">>]}).

Read the contexts guide for the query shapes, merges, working sets, slices, offline mode and errors.

API surface