macula_rag

Federated retrieval over the Macula mesh: ask every shard of an org, merge the answers.

This exists so an index split across many nodes can be searched as one, without a central server, and without any node the org did not delegate being able to answer for it.

What it does

A shard is one node holding part of an index. It registers a callback that searches its part, and publishes a summary of what it holds. A query asks every shard of the org at once and merges their hits by score.

The org decides who is a shard. Every shard provides the procedure <Org>/rag.query_shard_v1. macula 12 lets a node provide an org's procedure only with that org's D25 delegation, and lists a procedure's providers only after checking those delegations against the realm key. That checked list is the set of shards a query asks. A node the org did not delegate is never asked, whatever it publishes.
Scores are only merged where they compare. A shard's summary names the embedding its index was built with, model and dimension. A query asks only shards with the querying node's own embedding, and says which ones it skipped and why.
Nothing is dropped silently. A query answers {ok, Hits, #{answered => [NodeId], failed => [{NodeId, Reason}]}}. Every shard is in one list or the other: no_summary, {embedding_mismatch, Theirs}, timeout, malformed_answer, or the error the shard answered.
One shard per node. The node id is the shard's identity; shard_id is a label for people.

It needs macula 13.0.1 or later (~> 13.0.1): macula:providers/3,4 and the provider option of macula:call/6 arrived in 12.1, and a service on macula 13 cannot use a macula_rag pinned to 12.

Status

0.1.0, the first release. No node issues federated queries in production yet. What this version does not do:

The summary carries a bloom filter, and shards/0 lists it, but a query does not use it to choose shards: every trusted shard with a matching embedding is asked. Bloom-based selection is a later optimisation.
Trust is the org's delegation list. There is no further per-shard trust setting.

Using it

{deps, [{macula_rag, "~> 0.1"}]}.

Once a macula pool is connected:

ok = macula_rag:configure(Pool, RealmId,
       #{org => <<"acme">>,
         shard_id => <<"library-shard-1">>,
         realm_name => <<"io.macula">>,
         embedding => #{model => <<"nomic-embed-text">>, dim => 768}}),

%% This node is a shard: answer the org's queries.
ok = macula_rag:register_responder(
       fun(Query, #{top_k := K}) -> {ok, my_index:search(Query, K)} end),
ok = macula_rag:advertise([<<"library/books">>], Bloom),

%% Ask every shard of the org.
{ok, Hits, #{answered := Answered, failed := Failed}} =
    macula_rag:query(#{<<"text">> => <<"federated search">>}, #{top_k => 10}).

A hit needs at least a binary id and a numeric score; the rest of the map is yours and is carried as it is. Hits and queries arrive with binary keys, as every received payload does, and a merged hit also carries node_id and shard_id, the shard it came from.

realm_name must be the name whose SHA-256 is RealmId, or configure/3 refuses: the topics carry the name, and a topic naming another realm reaches nobody.

The grant

register_responder/1 advertises the org's procedure as soon as macula grants it, which needs the org's D25 procedure_delegation naming this node. Until then, status/0 says why not:

#{responder := {not_granted, #{reason := {provider_authorization,
                                          {procedure_delegation, not_found}},
                               since_ms := 42000}}}

{procedure_delegation, not_found} means an operator has to delegate this node for the org. The grant is asked for again every grant_retry_ms (30 s), so a delegation made later is picked up without a restart, and advertised once. A link that drops and comes back needs nothing: macula replays the advertisement on it.

Configuration

Option	Default	Meaning
`org`	required	whose shards these are; the procedure is `<org>/rag.query_shard_v1`
`shard_id`	required	a label for this node's shard
`realm_name`	required	the realm's name; its SHA-256 must be the realm id
`embedding`	required	`#{model => binary(), dim => pos_integer()}` this node's index uses
`query_timeout_ms`	1500	how long a query waits for the slowest shard
`grant_retry_ms`	30000	how often a refused grant is asked for again
`summary_republish_ms`	60000	how often this shard's summary is republished; a summary unheard for three periods is stale

The wire contract


procedure	`<org>/rag.query_shard_v1`
fact	`<realm>/<org>/rag/shard/shard_summarized_v1`: `shard_id`, `model`, `dim`, `topics`, `bloom`, `at_ms`
fact	`<realm>/<org>/rag/shard/shard_withdrawn_v1`: `shard_id`, `at_ms`
query	`query`, `top_k`, `model`, `dim`
answer	`shard_id`, `model`, `dim`, `hits`

A summary belongs to the node macula verified as its publisher, whatever the payload claims. test/macula_rag_contract_tests.erl pins every key and reads each payload back through macula's codec.

The RAG service contract

What any provider serves and any caller can check: provenance on every hit, the corpus hash naming which corpus answered, and the operator's optional signature over it. The RAG service contract is the canonical text, with frozen vectors in test/vectors/.

macula_rag:corpus_hash(Description)                 %% the hash of a description you built
macula_rag:verify_corpus(Description, P, Profile)   %% {ok, {signed, P}} | {ok, unsigned} | {error, Why}
macula_rag:verify_hit(Hit)                          %% the text is what its provenance hashes

verify_corpus/3 counts a signature only from the provider the caller pinned (macula:call/6 with #{provider => P}), derived from the verified key, never from signed_by.

Build and test

scripts/check.sh      # compile, eunit, lint, xref, dialyzer, as CI runs them

OTP 28.4.3, pinned in .tool-versions and CI. The suite includes a federation test that starts three shard VMs with peer: two answer one query and their hits merge by score; the third, built with another embedding, is reported and never asked.

Releasing

A pushed vX.Y.Z tag publishes to hex (.github/workflows/publish-hex.yml). Before publishing, the workflow refuses anything but the clean, pushed tag whose version is in src/macula_rag.app.src and has a CHANGELOG section, dry-runs the publish, and checks that hex.pm accepts the key. After publishing it checks that hex serves the tagged files.

Guides

License

Apache-2.0.