barrel_vectordb

Quick Start

With Embeddings

%% Start a store with local Python embeddings
{ok, _} = barrel_vectordb:start_link(#{
    name => my_store,
    path => "/tmp/vectors",
    embedder => {local, #{}}  %% requires Python + sentence-transformers
}).

%% Add documents (text is embedded automatically)
ok = barrel_vectordb:add(my_store, <<"doc1">>, <<"Hello world">>, #{}).
ok = barrel_vectordb:add(my_store, <<"doc2">>, <<"Goodbye world">>, #{}).

%% Search with text query
{ok, Results} = barrel_vectordb:search(my_store, <<"hi there">>, #{k => 5}).
%% => [#{key => <<"doc1">>, text => <<"Hello world">>, score => 0.89, ...}, ...]

Vector-Only (no embedder)

%% Start a store without embedder
{ok, _} = barrel_vectordb:start_link(#{
    name => my_store,
    path => "/tmp/vectors",
    dimensions => 768
}).

%% Add with pre-computed vectors
ok = barrel_vectordb:add_vector(my_store, <<"doc1">>, <<"Hello">>, #{}, Vector).

%% Search with vector query
{ok, Results} = barrel_vectordb:search_vector(my_store, QueryVector, #{k => 5}).

Installation

Add to your rebar.config:

{deps, [
    {barrel_vectordb, "2.1.1"}
]}.

This includes barrel_embed for embedding support. To use text-based operations (add/4, search/3), configure an embedder provider. Without an embedder configured, use add_vector/5 and search_vector/3 with pre-computed vectors.

Optional: Reranking

For cross-encoder reranking, add barrel_rerank:

{deps, [
    {barrel_vectordb, "2.1.1"},
    {barrel_rerank, "1.0.0"}
]}.

Core API

Add Documents

%% Add with text (requires embedder)
ok = barrel_vectordb:add(Store, Id, Text, Metadata).

%% Add with explicit vector (no embedder required)
ok = barrel_vectordb:add_vector(Store, Id, Text, Metadata, Vector).

%% Add batch (requires embedder)
{ok, #{inserted := N}} = barrel_vectordb:add_batch(Store, [
    {<<"id1">>, <<"text 1">>, #{type => a}},
    {<<"id2">>, <<"text 2">>, #{type => b}}
]).

Search

%% Search with text query (requires embedder)
{ok, Results} = barrel_vectordb:search(Store, <<"query text">>, #{k => 10}).

%% Search with vector (no embedder required)
{ok, Results} = barrel_vectordb:search_vector(Store, Vector, #{k => 10}).

%% Search with metadata filter
{ok, Results} = barrel_vectordb:search(Store, <<"query">>, #{
    k => 10,
    filter => fun(Meta) -> maps:get(type, Meta) =:= important end
}).

%% Search with optimized options (skip text/metadata for faster results)
{ok, Results} = barrel_vectordb:search_vector(Store, Vector, #{
    k => 50,
    include_text => false,      %% Skip text lookup
    include_metadata => false   %% Skip metadata lookup
}).

%% Search with custom ef_search (higher = better recall, slower)
{ok, Results} = barrel_vectordb:search_vector(Store, Vector, #{
    k => 10,
    ef_search => 200   %% Default is max(k, 50)
}).

Document Operations

%% Get document by ID
{ok, Doc} = barrel_vectordb:get(Store, <<"doc1">>).

%% Update document (requires embedder)
ok = barrel_vectordb:update(Store, <<"doc1">>, <<"New text">>, #{}).

%% Upsert (requires embedder)
ok = barrel_vectordb:upsert(Store, <<"doc1">>, <<"Text">>, #{}).

%% Delete
ok = barrel_vectordb:delete(Store, <<"doc1">>).

%% Peek (sample documents)
{ok, Docs} = barrel_vectordb:peek(Store, 10).

%% Count
N = barrel_vectordb:count(Store).

%% Checkpoint HNSW index (speeds up restart)
ok = barrel_vectordb:checkpoint(Store).

Configuration

barrel_vectordb:start_link(#{
    name => my_store,              %% Store name (required)
    path => "/var/data/vectors",   %% RocksDB path
    dimensions => 768,             %% Vector dimensions (default: 768)
    backend => hnsw,               %% Index backend: hnsw (default) or faiss
    embedder => EmbedderConfig,    %% Embedding provider (optional)
    hnsw => #{                     %% HNSW index parameters
        m => 16,
        ef_construction => 200
    },
    batch => #{                    %% Write batching options
        min_batch_size => 4,       %% Min requests before batching
        max_batch_size => 256      %% Max batch size
    }
}).

Index Backends

barrel_vectordb supports two vector index backends:

HNSW (Default)

Pure Erlang HNSW implementation. No external dependencies.

{ok, _} = barrel_vectordb:start_link(#{
    name => my_store,
    path => "/tmp/vectors",
    backend => hnsw  %% default, can be omitted
}).

FAISS

High-performance FAISS backend via barrel_faiss NIF. Typically 2-6x faster than pure Erlang HNSW for insert and search operations.

Installation:

Add to your rebar.config:

{profiles, [
    {faiss, [
        {deps, [
            {barrel_faiss, {git, "https://github.com/barrel-db/barrel.git", {branch, "main"}}}
        ]}
    ]}
]}.

Requires FAISS library installed on your system. See barrel_faiss README for installation instructions.

Usage:

{ok, _} = barrel_vectordb:start_link(#{
    name => my_store,
    path => "/tmp/vectors",
    backend => faiss,
    faiss => #{
        index_type => <<"HNSW32">>,  %% default
        distance_fn => cosine        %% cosine (default) or euclidean
    }
}).

Backend Comparison:

When to use FAISS:

• Large indexes (>100K vectors)
• High insert throughput requirements
• Search latency is critical

When to use HNSW:

• Simpler deployment (no NIF)
• Frequent deletions
• Smaller indexes

Vector Quantization

Reduce memory usage with TurboQuant compression:

%% Create quantizer (no training needed)
{ok, TQ} = barrel_vectordb_turboquant:new(#{
    dimension => 768,
    bits => 3
}).

%% Encode vector (768 floats -> ~388 bytes)
Code = barrel_vectordb_turboquant:encode(TQ, Vector).

%% Fast distance computation
Tables = barrel_vectordb_turboquant:precompute_tables(TQ, Query),
Distance = barrel_vectordb_turboquant:distance_nif(Tables, Code).

For large dimensions, use Subspace-TurboQuant:

{ok, TQS} = barrel_vectordb_turboquant_subspace:new(#{
    dimension => 1536,
    m => 16  %% 16 subspaces
}).

See TurboQuant Documentation for details.

Embedding Providers

Embedder is explicit - if not configured, only add_vector/5 and search_vector/3 work. Text-based operations return {error, embedder_not_configured}.

Local

Local Python with sentence-transformers. CPU-based, no external API calls.

embedder => {local, #{
    python => "python3",                %% Python executable (default)
    model => "BAAI/bge-base-en-v1.5",   %% Model name (default, 768 dims)
    timeout => 120000                   %% Timeout in ms (default)
}}

Setup with virtual environment (recommended):

# Create virtual environment
python3 -m venv ~/.venv/barrel_embed
source ~/.venv/barrel_embed/bin/activate

# Install dependencies
pip install sentence-transformers

# Verify installation
python -c "from sentence_transformers import SentenceTransformer; print('OK')"

Then start the store with the virtual environment's Python path:

{ok, _} = barrel_vectordb:start_link(#{
    name => my_store,
    path => "/tmp/vectors",
    embedder => {local, #{
        python => "/home/user/.venv/barrel_embed/bin/python"
    }}
}).

Or activate the venv before starting your Erlang application:

source ~/.venv/barrel_embed/bin/activate
rebar3 shell

%% Now python3 will use the venv automatically
{ok, _} = barrel_vectordb:start_link(#{
    name => my_store,
    path => "/tmp/vectors",
    embedder => {local, #{}}  %% uses default python3
}).

Supported Models:

Any sentence-transformers or HuggingFace model works. Popular choices:

The dimension is auto-detected from the model.

Ollama

Local Ollama server. Requires Ollama to be running.

embedder => {ollama, #{
    url => <<"http://localhost:11434">>,   %% Ollama API URL (default)
    model => <<"nomic-embed-text">>,       %% Model name (default, 768 dims)
    timeout => 30000                       %% Timeout in ms (default)
}}

# Pull embedding models:
ollama pull nomic-embed-text

Supported Models:

OpenAI

OpenAI Embeddings API. Requires an API key.

embedder => {openai, #{
    api_key => <<"sk-...">>,               %% API key (or set OPENAI_API_KEY env var)
    model => <<"text-embedding-3-small">>, %% Model name (default, 1536 dims)
    timeout => 30000                       %% Timeout in ms (default)
}}

# Set API key as environment variable (alternative to config)
export OPENAI_API_KEY=sk-...

Supported Models:

FastEmbed

Lightweight ONNX-based embeddings. Faster than sentence-transformers for many models.

embedder => {fastembed, #{
    python => "python3",                    %% Python executable (default)
    model => "BAAI/bge-small-en-v1.5",      %% Model name (default, 384 dims)
    timeout => 120000                       %% Timeout in ms (default)
}}

Setup:

pip install fastembed

Supported Models:

Provider Chain

Try providers in order until one succeeds.

embedder => [
    {openai, #{api_key => <<"sk-...">>}},  %% Try OpenAI first
    {ollama, #{url => <<"http://localhost:11434">>}},
    {local, #{}}  %% Fallback to CPU
]

Advanced Embedding Types

SPLADE (Sparse Embeddings)

Neural sparse embeddings with term expansion. Produces sparse vectors for hybrid search.

%% Initialize SPLADE provider
{ok, State} = barrel_embed:init(#{
    embedder => {splade, #{
        model => "prithivida/Splade_PP_en_v1"
    }}
}).

%% Get sparse vectors directly
{ok, SparseVec} = barrel_embed_splade:embed_sparse(<<"query text">>, Config).
%% => #{indices => [1, 42, 156], values => [0.5, 0.3, 0.8]}

Setup:

pip install transformers torch

ColBERT (Late Interaction)

Multi-vector embeddings for fine-grained token-level matching.

%% Initialize ColBERT provider
{ok, State} = barrel_embed:init(#{
    embedder => {colbert, #{
        model => "colbert-ir/colbertv2.0"
    }}
}).

%% Get multi-vector embeddings
{ok, MultiVec} = barrel_embed_colbert:embed_multi(<<"query text">>, Config).
%% => [[0.1, 0.2, ...], [0.3, 0.4, ...], ...]  %% One vector per token

%% MaxSim scoring between query and document
Score = barrel_embed_colbert:maxsim_score(QueryVecs, DocVecs).

Setup:

pip install transformers torch

CLIP (Image Embeddings)

Cross-modal embeddings for image-text search. Images and text share the same vector space.

%% Initialize CLIP provider
{ok, State} = barrel_embed:init(#{
    embedder => {clip, #{
        model => "openai/clip-vit-base-patch32"
    }}
}).

%% Embed text (for cross-modal search)
{ok, TextVec} = barrel_embed_clip:embed(<<"a photo of a cat">>, Config).

%% Embed image (base64 encoded)
{ok, ImageVec} = barrel_embed_clip:embed_image(Base64Image, Config).

%% TextVec and ImageVec are in the same space - compare with cosine similarity!

Setup:

pip install transformers torch pillow

Supported Models:

Reranking

Requires: barrel_rerank dependency

Cross-encoder reranking for improved search relevance. Use after initial vector search.

%% Add barrel_rerank to your deps
{deps, [
    {barrel_vectordb, "2.1.1"},
    {barrel_embed, "2.3.0"},
    {barrel_rerank, "1.0.0"}
]}.

%% Start the reranker
{ok, Reranker} = barrel_rerank:start_link(#{
    model => "cross-encoder/ms-marco-MiniLM-L-6-v2"
}).

%% Two-stage retrieval
%% Stage 1: Fast vector search (top 100)
{ok, Candidates} = barrel_vectordb:search(Store, Query, #{k => 100}).

%% Stage 2: Rerank candidates
Docs = [maps:get(text, C) || C <- Candidates],
{ok, Ranked} = barrel_rerank:rerank(Reranker, Query, Docs).
%% => [{0, 0.95}, {2, 0.82}, {1, 0.45}, ...]  %% {Index, Score}

%% Get top 10 after reranking
Top10 = [lists:nth(Idx + 1, Candidates) || {Idx, _} <- lists:sublist(Ranked, 10)].

%% Cleanup
ok = barrel_rerank:stop(Reranker).

Setup:

The venv with dependencies is auto-created on first use, or manually:

{ok, _} = barrel_rerank_venv:ensure_venv().

Supported Models:

BM25 Sparse Retrieval

Pure Erlang BM25 implementation for lexical search. In-memory index.

%% Create index
Index = barrel_vectordb_bm25:new().

%% Add documents
Index1 = barrel_vectordb_bm25:add(Index, <<"doc1">>, <<"The quick brown fox">>).
Index2 = barrel_vectordb_bm25:add(Index1, <<"doc2">>, <<"The lazy dog">>).

%% Search
Results = barrel_vectordb_bm25:search(Index2, <<"quick fox">>, 10).
%% => [{<<"doc1">>, 2.45}, ...]

%% Get sparse vector for a document
{ok, SparseVec} = barrel_vectordb_bm25:get_vector(Index2, <<"doc1">>).
%% => #{indices => [hash1, hash2, ...], values => [1.2, 0.8, ...]}

%% Index stats
Stats = barrel_vectordb_bm25:stats(Index2).
%% => #{doc_count => 2, avg_doc_len => 3.5, ...}

Note: BM25 index is in-memory and not persisted. Rebuild from documents on startup.

Search Options

HNSW Parameters

Testing

Unit Tests

Unit tests use mocking and don't require external dependencies:

rebar3 eunit

With Optional Dependencies

To run tests that exercise barrel_embed:

rebar3 as test_embed eunit

To run tests with all optional dependencies:

rebar3 as test_full eunit

Performance

Search Latency

Optimizations

• Batch writes: Concurrent writes are automatically batched via gen_batch_server
• Batch lookups: Search uses rocksdb:multi_get for efficient result fetching
• Skip options: Use include_text => false to skip unnecessary RocksDB reads
• HNSW optimization: O(log N) candidate management with balanced trees

Benchmarking

Run the benchmark suite:

rebar3 as bench compile && rebar3 as bench eunit --module=barrel_vectordb_bench

Backend Comparison Benchmarks

Compare HNSW vs FAISS performance:

# Quick comparison
./scripts/run_backend_bench.sh --quick

# Default comparison
./scripts/run_backend_bench.sh

# Full benchmark suite
./scripts/run_backend_bench.sh --full

Or programmatically:

rebar3 as bench_faiss shell
barrel_vectordb_backend_bench:run_all().

Architecture

• Storage: RocksDB with column families
• Index: HNSW for approximate nearest neighbor search
• Vectors: 8-bit quantization with norm caching
• Embeddings: Pluggable providers with fallback
• Batching: gen_batch_server for automatic write coalescing

See the API documentation for detailed architecture information.

Support

License

Apache-2.0. See LICENSE for details.

Built by Enki Multimedia | barrel-db.eu