barrel

The embeddable edge-AI database. barrel composes the document layer (barrel_docdb) and the vector layer (barrel_vectordb) behind one API, so an Erlang application can embed a single database that does documents, vectors, BM25, hybrid search, attachments (blobs), and a changes feed.

Documentation | HexDocs | Repository

A barrel database is a docdb database plus a vectordb store that share a name and a single id space: a document, its attachments (blobs), and its vector are all addressed by the same id. Blobs are docdb attachments; the storage backend is pluggable per database via the docdb barrel_att_backend seam (RocksDB BlobDB by default). barrel adds no storage of its own; it coordinates the layers. Each underlying app stays usable on its own.

Open and documents

{ok, Db} = barrel:open(mydb),

{ok, _}  = barrel:put_doc(Db, #{<<"id">> => <<"a">>, <<"title">> => <<"hello">>}),

{ok, Doc} = barrel:get_doc(Db, <<"a">>),

{ok, Rows, _Meta} = barrel:find(Db, #{where => [{path, [<<"title">>], <<"hello">>}]}),

ok = barrel:close(Db).

barrel:open/2 accepts #{docdb => Map, vectordb => Map} to pass options to each layer, including docdb => #{att_opts => #{backend => ...}} to choose an attachment backend.

Batches

[{ok, _}, {ok, _}] = barrel:put_docs(Db, [#{<<"id">> => <<"a">>}, #{<<"id">> => <<"b">>}]),

[{ok, _}, {ok, _}] = barrel:get_docs(Db, [<<"a">>, <<"b">>]),

{ok, #{inserted := 2}} = barrel:vector_add_batch(Db, [

    {<<"a">>, <<"t1">>, #{}, V1},

    {<<"b">>, <<"t2">>, #{}, V2}

]).

vector_add_batch/2 takes {Id, Text, Metadata} (text embedded by the store) or {Id, Text, Metadata, Vector} (explicit) tuples; a batch must be all one shape.

Vectors and search

ok = barrel:vector_add(Db, <<"a">>, <<"hello world">>, #{}, [0.1, 0.2, 0.3]),

{ok, Hits} = barrel:search_vector(Db, [0.1, 0.2, 0.3], #{k => 5}),

{ok, Hits2} = barrel:search_hybrid(Db, <<"hello">>, #{k => 5}).

Attachments

{ok, _} = barrel:put_attachment(Db, <<"a">>, <<"file.txt">>, <<"bytes">>),

{ok, <<"bytes">>} = barrel:get_attachment(Db, <<"a">>, <<"file.txt">>),

[<<"file.txt">>] = barrel:list_attachments(Db, <<"a">>).

Large attachments stream: open_attachment_writer/4 + write_attachment/2 + finish_attachment/1, and open_attachment_reader/3 + read_attachment/1.

Changes

{ok, Changes, LastHlc} = barrel:changes(Db, first),

{ok, StreamPid} = barrel:subscribe(Db, LastHlc).

API surface