mcl-om

Over-mesh substrate for the PQ fleet: the shared library every macula-services/mcl-X service daemon stands on, built on macula 12 ({macula, "~> 12.0"}), post-quantum only. The mcl-* services it carries replace the obsolete hecate-* services one port at a time.

Services in this org are edge-first. A service runs wherever its operator puts it (a cooperative infrastructure node, a relay box, a lab machine, a laptop) and dials out to a macula-station over QUIC. It needs no inbound port, no public address and nothing underneath it: the station it dials is what puts it on the mesh, and that is how a service on one edge box reaches a service on another.

Placement is therefore a deployment decision, not a property of the substrate. What a service always carries with it is its own node identity: a puzzle-hardened pq_hybrid node key whose node id derives from the carried public key. It answers as itself, never as the human whose machine it happens to be running on. See guides/identity_model.md for the town/library metaphor that drives the identity choices.

                         mcl-om
                            │
        ┌──────────┬────────┼─────────┐
        ▼          ▼        ▼         ▼
    mcl-echo  mcl-warden mcl-sentinel …

Every service is a separate OTP release shipped as an OCI container to ghcr.io/macula-services/. mcl-om is the library they all link against to behave consistently on the mesh: the same service contract, the same health endpoint, the same identity-claim flow, the same capability-advertise pattern, and the same generated repository (Containerfile, compose file, CI workflows).

What this library is (and isn't)

It is:

An Erlang behaviour (mcl_om_service) — six callbacks every service implements: start/1, stop/1, health/0, capabilities/0, identity_spec/0, info/0.
Helpers for the bits every service needs: load (or generate) the puzzle-hardened node key, advertise a capability on the mesh as a signed DHT record (callable when it carries a handler), serve a /health endpoint.
The mcl_service rebar3 template for a new service repository: application, supervisor, service module with its eunit suite, release config, Containerfile, compose.yml, health.sh and both CI workflows. See Scaffold a new service.

It is not:

A daemon. It has no application:start_phase of its own beyond the library's facade.
A plugin host. Services are containerised.
A network library. Services talk to a PQ macula-station via the macula 12 SDK like any other Macula client: outbound only. The station does the peering, the DHT and the routing, which is what lets a service sit behind NAT at the edge and still be reachable.

Layering position

Layer 4 — apps        user-facing apps

Layer 3 — session     per-identity sessions and UI

Layer 2 — services    macula-services/mcl-echo, -warden, -sentinel, …
                      Always-on, containerised, system-class workloads,
                      each with its own node identity.
                      Run at the edge or on realm infrastructure; either
                      way they dial out to a station.
                      ↑↑↑ this library is the substrate ↑↑↑

Layer 1 — identity    macula-realm

Layer 0 — kernel      macula-station (the PQ fleet)

See philosophy/HECATE_TIER_MODEL.md in hecate-corpus for the longer cut-criteria discussion. Note that the tier model still phrases the L2 placement rule absolutely ("NOT on user laptops"); read that as a policy about where the realm's own shared services belong, not as a limit on what an mcl-om service can do.

Service ≠ provider. A node that serves a procedure is not thereby a service: every SDK lets a client serve procedures (pool.Serve in macula-go, say). A service is a node under this contract — always-on, containerised, a claimed identity answering Org/info and /health, advertised capabilities, and placement through macula-fleet. For mesh services the org runs BEAM only, and this library is the substrate; for mesh clients the SDKs are enough. A daemon written in another stack that wants onto the fleet under this contract is ported to Erlang/mcl-om, or runs outside the fleet's responsibility.

The contract

-module(my_service).
-behaviour(mcl_om_service).

%% lifecycle
-export([start/1, stop/1]).

%% introspection
-export([health/0, capabilities/0, identity_spec/0, info/0]).

start(_Opts) ->
    my_service_sup:start_link().

stop(_State) ->
    ok.

%% Reported on /health endpoint. Return ok | {degraded, Reason} | {down, Reason}.
health() ->
    ok.

%% Advertised onto the mesh via mcl_om_capabilities:advertise/1.
%% Other services / plugins find you by these.
capabilities() ->
    [
        #{name => <<"my_service.do_thing">>, version => 1},
        #{name => <<"my_service.list_things">>, version => 1}
    ].

%% The authority this service asks the realm for, and deliberately nothing
%% more. Ask for exactly the topics you publish and subscribe to.
identity_spec() ->
    #{
        scope     => <<"my_service">>,
        actions   => [<<"publish_summary">>, <<"answer_query">>],
        resources => [<<"my_service/*">>],
        ttl_days  => 30
    }.

info() ->
    #{
        name        => <<"mcl-my-service">>,
        version     => <<"0.1.0">>,
        description => <<"What this service does in one line">>
    }.

That's the whole user-side contract. Six small functions. Health endpoint wiring and mesh advertisement come from mcl-om; the release, container image, compose file and CI workflows come from the mcl_service template described below.

On the wire every procedure is Org/Name, with the org from mcl_om's org app env; a service without a usable org refuses to boot. What every service gets without writing it:

Org/info, open to any mesh caller: name, version, description, the claim labels (service_name, box), org, node id, macula and mcl_om versions, uptime, the health word (ok, degraded, down, unknown, from the verdict /health last computed, never a fresh probe) and the procedures it advertises. No environment, paths, keys or reasons. It is also what makes a publish-only service count as online on the realm's Providers desk. A service may not declare its own info.
handler_timeout_ms on a capability, 1 to 600000: how long macula waits for the handler before answering the caller temporary_relay_failure (default 30000). Response capabilities only.
failed_publishes on /health: publishes whose publisher exited before resolving. mcl_om_pubsub runs each publisher under a watcher, so such an exit is counted and logged instead of killing the process that published.

Optional: store-backed services

CMD/PRJ services that own a reckon-db event store export three more optional callbacks. When a service exports store_id/0 + data_dir/0, mcl_om:boot/1 auto-starts the store and its evoq subscription before start/1 runs — you never call reckon_db_sup:start_store/1 yourself. Producer-only services (no store) omit these and pay nothing.

%% Optional store-wiring callbacks (only if the service owns a store)
-export([store_id/0, data_dir/0, store_indexes/0]).

%% Atom store id. Data lands at <data_dir>/<store_id>/.
store_id() -> my_service_store.

data_dir() -> "/var/lib/mcl-my-service".

%% reckon-db secondary index declarations installed on the store. This is
%% how CCC payload indexes get declared — without it the store starts with
%% no secondary indexes and payload/hash queries find nothing.
store_indexes() ->
    [tags, event_type,
     {payload, <<"plate">>},                            %% single-field index
     {payload_hash, [<<"lot_id">>, <<"plate">>]}].      %% composite hash index

store_indexes/0 is itself optional: export it only when the store needs secondary indexes. Omit it (or return []) for a store with none.

Scaffold a new service

# From the directory that will hold the new repository,
# typically ~/work/github.com/macula-services:
scripts/scaffold-service.sh mcl-newservice "Does X over the mesh" 8484

That is a thin wrapper over rebar3 new mcl_service, and it exists so you say the name once. A service has two names: the repository, the container image and the name it answers to on the mesh are kebab-case, while the OTP application and every module prefix are snake_case because they are Erlang atoms. Mustache has no functions, so a template cannot derive one from the other. The generated eunit suite asserts the two agree modulo the separator, so a hand-rolled rebar3 new with a mismatched pair still fails on its first test run.

To use rebar3 new directly, install the templates once. rebar3 only finds custom templates under ~/.config/rebar3/templates, and an empty directory has no dependencies to carry them there:

scripts/install-templates.sh          # symlinks; --remove to undo
rebar3 new mcl_service repo=mcl-newservice name=mcl_newservice \
    desc="Does X over the mesh" org=your-org registry=ghcr.io health_port=8484

The scaffold is not house-specific. org, registry, builder_image and runtime_image are variables, and nothing generated names our organisation, our registry, our images, our deployment repository or our hosts. If you are building a service for your own mesh, set those four and everything else follows. scaffold-service.sh defaults them to ours because that is who runs it most; MCL_ORG, MCL_REGISTRY, MCL_BUILDER_IMAGE and MCL_RUNTIME_IMAGE override. A test generates a service as a stranger and fails if any of our own specifics survive.

The image pair. builder_image is the image the release is built in and the one the generated lint job runs in; runtime_image is the one it runs on. Both are pinned by digest, and they are a pair: a release built in one runs on the other's libc and OpenSSL, so scaffold-service.sh refuses to override just one. The defaults are the fleet's plain pair from macula-io/macula-ci-images, the pair mcl-tube, mcl-sentinel, mcl-warden and most other mcl services build FROM. Not all of them: mcl-echo (and mcl-graph) still build on the retired hexpm Alpine OTP and run on Alpine, and the services with a rocksdb read model (mcl-stations, mcl-mail, mcl-rag) use the rocksdb pair.

Variable	Default	Must carry
`builder_image`	`ghcr.io/macula-io/macula-ci-otp:20260923-1444@sha256:dd2ba6eb…`	OTP 28.4.3 with an OpenSSL that has ML-DSA, rebar3, Rust, a C toolchain, git
`runtime_image`	`ghcr.io/macula-io/macula-pq-runtime:20260923-1444@sha256:15a5501b…`	the builder's libc, OpenSSL with ML-DSA, libstdc++, ncurses, curl

They are named once, as the defaults in priv/templates/mcl_service.template; the script passes an image only when one is overridden, and the template suite reads them from there. The generated Containerfile refuses to build on any OTP but 28.4.3 with mldsa87, and the generated lint job checks every tool in the table before it checks out the code. A service that links the erlang rocksdb binding (barrel_docdb) moves both lines to the rocksdb pair, macula-ci-otp-rocksdb and macula-pq-runtime-rocksdb. On the plain pair rocksdb's CMake silently disables any compression backend whose -dev package is missing, the build stays green, and barrel's default snappy blob compression then fails at db_open ("The specified blob compression type Snappy is not available").

Generates a repository that compiles, tests and deploys:

apps/<app>/src/ — the .app.src, _app.erl, _sup.erl and a _service.erl implementing the behaviour
apps/<app>/test/ — a suite asserting the contract's shape, the two names, and that the reported version is the application's own
rebar.config with a relx release, the prod profile and the elvis ruleset
config/sys.config.src and config/vm.args.src
Containerfile: built in builder_image, run on runtime_image
deploy/docker-compose.yml — the service's own run contract, not the deployed file; fleet placement lives in macula-io/macula-fleet. It mounts a named volume, <repo>-secrets, at /etc/mcl/secrets, where the service's identity key lives, so the node id survives a container recreate
.github/workflows/: lint (in builder_image) and build-push to registry
scripts/health.sh, executable
README.md, CHANGELOG.md, LICENSE, .gitignore

It emits no TODO and no stub. What it generates is honestly complete and empty: the supervisor has no children, and the service announces no capability and requests no authority. Those are the correct answers for a service that does nothing yet, and each is asserted by a generated test, so filling one in is a deliberate act that breaks a test rather than a comment someone forgets.

The templates are exercised by mcl_service_template_SUITE, which generates a service for real and compiles it. The suite exists because the previous templates drifted unnoticed for months, and a template with no test is documentation that compiles.

One thing it cannot do for you, and it has bitten: the registry package may be created private, and the pull then fails on the host with a bare unauthorized that names nothing. Check it after the first build.

Deploying on the BEAM Campus fleet

Ours, and deliberately not part of the scaffold. deploy/docker-compose.yml in a generated service carries what the service knows about itself; macula-io/macula-fleet carries placement: which box, which station pins, which realm key, which secret file. The boxes pull it; nothing is pushed to them.

How a box runs its services. Each box has macula-fleet checked out at ~/gitops/macula-fleet. The hecate-reconcile systemd --user timer runs edge/gitops/reconcile.sh every 2 minutes: it fast-forwards the checkout and runs docker compose up -d for every row of edge/<box>/reconcile.manifest. Who updates images is set per box in edge/<box>/reconcile.options. The default is watchtower, which polls every 60 s and rolls any container labelled com.centurylinklabs.watchtower.enable=true onto a new :latest; the reconciler then applies config only.

To put a new service on a box:

Add edge/scripts/docker-compose.<service>.yml to macula-fleet: the image, network_mode: host, the named identity volume at /etc/mcl/secrets, the watchtower label, and the environment the service reads.
Add a row to edge/<box>/reconcile.manifest: <project> <compose> <config-env|-> <secret-file|-> <prep|->. Public per-box configuration, such as MCL_REALM_KEY (the realm's public key), goes in a committed edge/<box>/<service>-config.env.
Seed the secrets once, on the box, at ~/.hecate/secrets/<name>.env, 0600, never committed (MCL_COOKIE, for one). A row whose secret file is missing starts nothing: the reconciler logs [reconcile] FAILED: <project>, and only journalctl --user -u hecate-reconcile shows it.
Push macula-fleet. The box picks it up on its next tick, within 2 minutes.

Health ports bound on beam00 today: 8450, 8461 (mcl-echo), 8484, 8494. Host networking turns a collision into a silent bind failure.

Status

Working library — 0.31.x, on macula 12.7. The behaviour and all helpers are implemented (mcl_om_identity, mcl_om_capabilities, mcl_om_store, mcl_om_health), the boot path (mcl_om:boot/1 with auto store-wiring) is exercised by a Common Test suite (mcl_om_SUITE), and rebar3 new mcl_service generates a service that compiles, tests and deploys, guarded by a suite that generates one for real. mcl_om:mesh_handles/0 gives every service the shared {Pool, Realm} pair its own PubSub/RPC/Content code needs, alongside realm/0 and identity_key/0 on the same public facade. Lint, EUnit and the Common Test suites run on every pull request and every push to main.

The behaviour surface has grown since the first cut: the store-wiring callbacks are store_id/0 + data_dir/0 (required together) plus optional store_indexes/0 (CCC secondary indexes), store_mode/0 (single | cluster), and store_integrity/0 (per-store HMAC event tamper-resistance). See the CHANGELOG for the evolution.

Known gap: the store-wiring callbacks are the part of the contract with no test of their own. mcl_om_SUITE boots a producer-only dummy service.

Consumers: the macula-services/mcl-* services, mcl-echo deployed on the fleet. Not yet burned in under sustained production load.

License

Apache-2.0. See LICENSE.