Macula SDK

Erlang/OTP client SDK for the Macula HTTP/3 mesh

Latest — 9.0.0 (breaking): LAN clustering and distribution-over-mesh split into independent concerns — macula_cluster_system/ vs. macula_dist_system/. macula_dist_relay is renamed macula_dist_pool (the facade, join_mesh/1 / join_dist_relay/1, is unaffected); the auto_cluster sys.config option is removed (start clustering explicitly via macula_cluster:start_cluster/1). See CHANGELOG.md for the full Breaking section. 8.10.0 added chunked content sharing; 8.9.0 added streaming direct-dial; 8.8.0 added per-publisher pubsub delivery ordering.

What is Macula?

Macula is an Erlang/OTP client SDK for building applications on a mesh of stations — realm-agnostic relays that route over QUIC (HTTP/3) and form a Kademlia DHT. Your service or daemon connects outbound to one or more stations: no open ports, NAT-friendly, no VPN. It provides:

• RPC (request/response) — discover a provider in the DHT, then dial its serving station directly (one hop), with optional realm-CA trust verification.
• Pub/Sub — topic-based event fan-out across stations, with per-publisher ordered delivery.
• Content — content-addressed sharing and live streaming (MCID).
• DHT records — signed, TTL'd records (advertisements, endpoints, more).
• Erlang distribution over mesh — net_adm:ping across firewalls, no VPN.
• Identity — Ed25519 keypairs, UCAN tokens, DID documents (NIF-accelerated).
• MRI — typed, hierarchical resource identifiers.
• Zero-config LAN clustering — UDP-multicast gossip.

The station (routing, DHT, SWIM, peering) is a separate repo, macula-station; this package is the client you build against.

Quick Start

Add to rebar.config:

{deps, [{macula, "~> 8.8"}]}.

Or in Elixir mix.exs:

defp deps do
  [{:macula, "~> 8.8"}]
end

application:ensure_all_started(macula),

%% Connect a pool to one or more stations (seed URLs). The pool owns one
%% QUIC link per seed, reconnecting and replaying subscriptions as needed.
{ok, Pool} = macula:connect([<<"quic://boot.macula.io:443">>], #{}),

%% A realm is a 32-byte tag derived from a name; it scopes every call.
Realm = macula_realm:id(<<"io.example.myapp">>),

%% Subscribe (delivers {macula_event, Ref, Topic, Payload, Meta} to a pid),
{ok, Ref} = macula:subscribe(Pool, Realm, <<"sensors.temperature">>, self()),

%% or subscribe with a callback fun(Topic, Payload, Meta):
{ok, Ref2} = macula:subscribe_callback(
    Pool, Realm, <<"sensors.temperature">>,
    fun(_Topic, Payload, _Meta) -> io:format("~p~n", [Payload]) end),

%% Publish. Entity IDs go in the PAYLOAD, never in the topic.
ok = macula:publish(Pool, Realm, <<"sensors.temperature">>,
                    #{sensor => <<"kitchen">>, value => 23.5}),

%% Advertise an RPC procedure (open to any identified caller here),
ok = macula:advertise(Pool, Realm, <<"math.add">>,
                      fun(#{<<"a">> := A, <<"b">> := B}) -> {ok, A + B} end,
                      #{}),

%% Call it — the SDK resolves the provider and dials its station directly.
{ok, 5} = macula:call(Pool, Realm, <<"math.add">>,
                      #{<<"a">> => 2, <<"b">> => 3}, 5_000).

The Four Interaction Patterns

Macula gives you four ways for two parties to interact over the mesh. The point-to-point ones — RPC, content, and streaming — share one shape: resolve in the DHT, then dial the serving station directly (one hop). RPC and streaming resolve+dial directly by default (call_station, call_stream_station); content sharing reaches a copy via the connected station's own relay by default and can dial a specific announced host directly (find_content_providers + call_station) when that is not enough. Pub/Sub is the deliberate exception: it fans out through the stations, because broadcasting to many interested parties is a different problem than a two-party exchange.

1. RPC — direct-dial

A provider publishes a signed procedure_advertisement naming its serving station. A consumer resolves it over the DHT, optionally verifies the provider's realm-issued cert chains to the realm CA (dropping squatters), resolves the serving station's endpoint, and dials it directly for the call. Discovery is O(log N) DHT lookups; the call itself is one hop.

%% High-level: the SDK composes resolve -> dial -> call for you.
{ok, Result} = macula:call(Pool, Realm, <<"math.add">>, Payload, 5_000),

%% Low-level: dial a known station URL yourself.
{ok, Result} = macula:call_station(Pool, <<"quic://station-b:443">>,
                                   Realm, <<"math.add">>, Payload, 5_000).

2. Pub/Sub — fan-out with per-publisher ordering

Subscription interest gossips across stations; a publish fans out along the Plumtree eager tree, with lazy IHAVE/GRAFT gossip repairing a missed message. One subscriber holds one connection and receives from every publisher, wherever they are. Topics name event types — energy.home.measured, not energy.home.42.measured — so IDs live in the payload and topics never explode.

Since 8.8, a single publisher's stream is delivered in order per publisher. Pick the contract at subscribe time:

%% ordered (default): per-publisher FIFO; a missing seq skipped after a timeout.
{ok, R1} = macula:subscribe(Pool, Realm, Topic, self()),

%% latest_only: newest-wins, drop stale, no head-of-line delay (state snapshots).
{ok, R2} = macula:subscribe(Pool, Realm, Topic, self(), #{delivery => latest_only}),

%% as_arrives: raw arrival order; the consumer orders it itself.
{ok, R3} = macula:subscribe(Pool, Realm, Topic, self(), #{delivery => as_arrives}).

See the PubSub Guide for the ordering model and tuning (order_timeout_ms, order_max_buffer, the pubsub_gap_skips telemetry).

3. Content Sharing — content-addressed

Content is addressed by an MCID — a hash of the bytes — so any host with the bytes serves the same MCID and integrity is self-verifying. Content that fits in one 256 KiB block round-trips as a single block (unchanged since v4.2.7); larger content is split into chunks and a Merkle-verified manifest, transparently. Chunked content is announced automatically, so a consumer can resolve every host serving an MCID and dial a specific one directly. See the Content Guide.

{ok, MCID}  = macula:put_content(Pool, Bytes),        %% any size
{ok, Bytes} = macula:get_content(Pool, MCID),
{ok, Hosts} = macula:find_content_providers(Pool, MCID).

4. Content Streaming — live QUIC stream

Same resolve-then-dial shape, but instead of a finite blob it opens an ordered QUIC stream: frames flow as produced, paced by QUIC per-stream flow control, ending when the source stops. Re-resolve on a stall — a listed source can be dead. Server-push, client-push, and bidirectional modes are supported; see the Streaming Guide.

%% resolve the source (find_records -> read_procedure_advertisement ->
%% station_endpoint, as in RPC), then dial its station directly
{ok, Stream} = macula:call_stream_station(Pool, StationUrl, Realm,
                                          <<"live.feed">>, Request, #{}),
loop_recv(Stream).  %% macula:recv/1 until eof

Erlang Distribution Over Mesh

Full OTP distribution tunneled through the mesh. No VPNs, no open ports.

macula:join_mesh(#{
    realm  => <<"io.macula">>,
    relays => [<<"quic://boot.macula.io:443">>],
    site   => #{name => <<"my-site">>, lat => 51.5, lng => -0.1}
}),
net_adm:ping('other@remote-host').  %% => pong

LAN Clustering

Zero-configuration cluster formation over UDP multicast:

ok = macula_cluster:start_cluster(#{strategy => gossip,
                                    secret => <<"my_cluster_secret">>}).

Identity and Crypto (NIF-accelerated)

Rust NIFs with pure-Erlang fallbacks:

%% Ed25519 keypair (a #{public := _, private := _} map)
KP  = macula_identity:generate(),
Sig = macula_identity:sign(<<"hello">>, KP),
true = macula_identity:verify(<<"hello">>, Sig, macula_identity:public(KP)),

%% BLAKE3 hashing
Hash = macula_blake3_nif:hash(Data),

%% UCAN capability tokens + DID documents
{ok, Token}   = macula_ucan_nif:create(Issuer, Audience, Caps, PrivKey),
{ok, Payload} = macula_ucan_nif:verify(Token, PubKey).

MRI (Resource Identifiers)

Typed, hierarchical resource addressing: mri:{type}:{realm}/{path}

{ok, Parsed} = macula_mri:parse(<<"mri:app:io.macula/acme/counter">>),
MRI = macula_mri:new_app(<<"io.macula">>, <<"acme">>, <<"counter">>),

%% Trie index for million-scale hierarchy queries
{ok, Idx}      = macula_mri:build_index(MRIs),
{ok, Children} = macula_mri:index_children(Idx, <<"io.macula">>, [<<"acme">>]).

Documentation

The station server lives in macula-station.

Related Projects

License

Apache 2.0 — see LICENSE.

_{Built with the BEAM}