libcluster_mesh

libcluster strategies for mesh VPNs that discover peers by polling the local mesh daemon — no cloud API, no API keys, no static host lists.

Four strategies are included:

Because peers are read from the node's own daemon, these strategies work with self-hosted control servers where cloud-API-based strategies (such as libcluster_tailscale) cannot: the daemon already holds the full netmap, including each peer's mesh IP, hostname, and online state. Node discovery generates no traffic outside the host, and distribution traffic flows only over the encrypted mesh.

Installation

def deps do
  [
    {:libcluster_mesh, "~> 0.1"}
  ]
end

Requires Elixir 1.18+, for the built-in JSON module. That is the only requirement it adds: Elixir 1.18 already requires OTP 25+, which is well past the OTP 23.1 that made -erl_epmd_port work, and the EPMD-less setup below needs both that and -start_epmd false.

Prerequisites

This library only does peer discovery. Joining the mesh — installing the daemon, authenticating, connecting — is out of scope and must be completed on every node before the strategies are of any use:

1. The node is a connected mesh member. Tailscale: tailscale up (browser login or --authkey tskey-auth-…). NetBird: netbird up --setup-key …. ZTL: ztlctl login --controller-url … --controller-auth-key … followed by ztlctl up. Plain WireGuard: the interface is configured with keys and peers. Authentication state is persisted by the daemon, so this is a one-time provisioning step per node — bake it into cloud-init, your container entrypoint, or whatever provisions the machine. The library itself never touches mesh credentials.

2. The daemon and its CLI are present. Tailscale: tailscaled running and the tailscale CLI on $PATH (or set cli:). NetBird: the netbird daemon running and its CLI on $PATH. ZTL: ztld running in TUN mode and ztlctl on $PATH (or set cli: / socket:). WireGuard: the interface up and wg on $PATH.

3. The BEAM's OS user can query the daemon. Discovery goes over the daemon's local socket, so its file permissions are the access control: for Tailscale on Linux, root or the user named in tailscale set --operator=<user>; for ZTL, the socket owner (root mode: 0660 root, userspace mode: the owning user); for NetBird and wg show, typically root.

4. The node's own mesh hostname matches what the others look for. On the named meshes, hostnames: decides which peers this node will dial. It does not decide who may dial this one, and nothing here asks the mesh to advertise a particular name — the daemon does that, from the machine's hostname, when it joins:

   tailscale up --hostname=myapp-1
   netbird up --hostname myapp-1 --setup-key …
   ztlctl set --hostname myapp-1
   

   A daemon may take the machine's hostname instead, and keep the name it registered with — in which case the command above reports success and changes nothing. Check the daemon's own status rather than its exit code. Where that is what it does, the machine's hostname has to be right before the daemon first registers.

   Erlang distribution is symmetric, so a node with a name nobody recognises still joins the cluster if it dials someone who is in it. That is why getting this wrong often looks like it works. It stops working when no node has a matching name — a fleet of embedded devices all still called nerves-a1b2, say — and then nothing dials anything, forever, and no error is logged because an empty peer list is what a quiet mesh looks like.

Once a node satisfies these, starting the application is all that remains — discovery and clustering are automatic from there.

Security model

The Erlang cookie is the authentication. Nothing else here is.

hostnames and cidr are discovery filters: they decide which of the daemon's peers are worth dialling, so a strategy does not spend every poll connecting to laptops and phones. They are not a trust boundary. Any device on the mesh that calls itself myapp-evil passes the prefix test, and any address inside the CIDR passes that one. What stops it is the cookie it does not have.

So: give every node the same cookie, keep it out of source, and treat anything that can reach the distribution port as able to try. Two things follow from that being true —

• Distribution traffic is not encrypted by this library. It is encrypted because the mesh encrypts everything, which is the point of running over one. Bind the distribution listener to the mesh interface (inet_dist_use_interface) so a node with another interface up is not reachable on it.
• Revoking a device from the mesh does not immediately eject it from the cluster. An established connection lives until net_ticktime notices it is gone — 60 seconds by default.

Nerves

These strategies run on Nerves. The prerequisites above are still the whole of the work, but a Nerves device supplies none of them for you, and four things about the platform shape how you meet them: the root filesystem is read-only, there is no init system, the VM boots without distribution, and the kernel module the mesh needs is present but not loaded.

Picking a system

Every official system for real hardware ships CONFIG_TUN=m and CONFIG_WIREGUARD=m — rpi0 through rpi5, bbb, grisp2, trellis — so tun.ko and wireguard.ko are in the image and no custom system is needed. nerves_system_x86_64 has neither; on that target everything below requires building a system of your own first.

What the images do not carry is iptables, iproute2 and wg. The first two are not missed: Tailscale-derived daemons configure their interface and routes over netlink. wg is another matter — it is the entirety of Cluster.Strategy.WireGuard's input, and no official system sets BR2_PACKAGE_WIREGUARD_TOOLS. Point cli: at the wg that vintage_net_wireguard builds into its priv, or enable the package in a custom system. The other three strategies need nothing that is not already in the image.

Putting the daemon in the firmware

These meshes distribute static Go binaries, which makes this the easy part: drop the daemon and its CLI into the overlay and they are in the image with their execute bits intact.

rootfs_overlay/usr/bin/ztld      # or tailscaled, netbird
rootfs_overlay/usr/bin/ztlctl    # the CLI the strategy polls

# mix.exs
{:libcluster_mesh, "~> 0.1"},
{:muontrap, "~> 1.6"}

Give the strategy the absolute path with cli: rather than relying on $PATH.

Four things then differ from starting the same daemon on an ordinary host:

Load tun first. The module is in the image but nothing has loaded it, and /dev/net/tun does not exist until something does. Tailscale-derived daemons run /sbin/modprobe themselves and busybox puts modprobe exactly there, so this is belt-and-braces — but it turns a confusing daemon failure into an obvious one.

System.cmd("/sbin/modprobe", ["tun"])

Put the daemon's state under /data. Defaults point at /var/lib, which is read-only. /data is the writable partition and survives firmware updates. Every daemon has a flag for this and they disagree on its name (tailscaled --statedir, ztld --state-dir).

Turn the daemon's DNS handling off. By default these daemons rewrite /etc/resolv.conf, which on Nerves is a symlink into /tmp owned by vintage_net. The attempt fails, and it takes the daemon's whole backend with it — reported as nothing more specific than a backend that would not initialize. ztlctl set --dns-mode none, tailscaled --no-dns.

Do not use userspace networking. It looks like the right choice for an embedded system and it is not: in that mode outbound traffic only leaves through a SOCKS5 proxy, so :gen_tcp.connect to a peer's mesh address times out and Erlang distribution has no way through. In kernel TUN mode the same connections succeed, and the module is already in the image.

Supervise it with MuonTrap. A plain Port leaves the OS process alive when the BEAM process that owns it dies. There is no init system here; the supervision tree is it.

How the daemon authenticates is its own business and unchanged by Nerves: an auth key for unattended joins, or the interactive flow where up prints a URL for someone to authorize the device at. Worth knowing only that a key reached through config :nerves, :erlinit, env: … is baked into the image, and so is shared by every device flashed with that firmware.

Distribution

Nerves boots the VM undistributed, and the node has to be named for the address the mesh assigns — which nothing knows at boot. So distribution starts from your code, after the daemon reports an address, and Cluster.Supervisor starts after that: a strategy that comes up before the node is named has nothing to connect from.

EPMD is not running either. Take the EPMD-less route described under Usage and put the flags in rel/vm.args.eex, among the others rather than at the end of the file — as its comments explain, everything after its -extra line is passed on as a plain argument instead of to the emulator.

-start_epmd false
-erl_epmd_port 45892

:init.get_argument(:start_epmd) answers {:ok, [[~c"false"]]} once the VM has them. If you would rather keep EPMD, it is inside the release's own ERTS and starting it is enough:

System.cmd("/srv/erlang/erts-#{:erlang.system_info(:version)}/bin/epmd", ["-daemon"])

All of it together

One child of your supervision tree owns the sequence. It is a chain of handle_continue/2 because each step needs the one before it: no address until the daemon has connected, no node name until there is an address, nothing for a strategy to connect from until the node has a name.

defmodule MyApp.Mesh do
  use GenServer
  require Logger

  @sock "/run/lolipopztl/ztld.sock"
  @state_dir "/data/lolipopztl"

  def start_link(opts), do: GenServer.start_link(__MODULE__, opts, name: __MODULE__)

  @impl true
  def init(_opts), do: {:ok, %{}, {:continue, :daemon}}

  @impl true
  def handle_continue(:daemon, state) do
    File.mkdir_p!(@state_dir)
    File.mkdir_p!(Path.dirname(@sock))
    # Belt and braces, so it must not be the thing that stops the node: the
    # daemon loads this itself, and a system without modprobe is one where it
    # is built in.
    if File.exists?("/sbin/modprobe"), do: System.cmd("/sbin/modprobe", ["tun"])

    {:ok, _} =
      MuonTrap.Daemon.start_link(
        "/usr/bin/ztld",
        ["--state-dir=#{@state_dir}", "--socket=#{@sock}"],
        log_output: :info,
        name: :ztld
      )

    {:noreply, state, {:continue, :join}}
  end

  def handle_continue(:join, state) do
    wait_for(fn -> File.exists?(@sock) end)

    # /etc is read-only, and the daemon takes its whole backend down rather
    # than carrying on without being able to write resolv.conf.
    ztlctl(["set", "--dns-mode", "none"])
    ztlctl(["login", "--controller-url", url(), "--controller-auth-key", key()])
    ztlctl(["up"])

    {:noreply, state, {:continue, {:cluster, wait_for(&address/0)}}}
  end

  def handle_continue({:cluster, ip}, state) do
    # The EPMD-less flags in rel/vm.args.eex are what make this succeed;
    # without them it fails at :nodistribution.
    {:ok, _} = :net_kernel.start([:"myapp@#{ip}", :longnames])
    Node.set_cookie(String.to_atom(System.fetch_env!("CLUSTER_COOKIE")))

    topologies = [
      mesh: [
        strategy: Cluster.Strategy.LolipopZTL,
        config: [
          node_basename: "myapp",
          hostnames: ["myapp-"],
          cli: "/usr/bin/ztlctl",
          socket: @sock
        ]
      ]
    ]

    {:ok, _} = Cluster.Supervisor.start_link([topologies, [name: MyApp.Cluster]])
    Logger.info("clustering as #{Node.self()}")
    {:noreply, state}
  end

  defp ztlctl(args) do
    System.cmd("/usr/bin/ztlctl", ["--socket=#{@sock}" | args], stderr_to_stdout: true)
  end

  defp address do
    {out, _} = ztlctl(["status", "--no-peers"])

    if out =~ "Connected" do
      Regex.run(~r/^ip:\s*(\S+)/m, out, capture: :all_but_first) |> List.first()
    end
  end

  # The daemon is talking to a controller over the network, so every step of
  # this takes as long as it takes. Raising rather than returning nil: the
  # supervisor restarts this and the reason is in the log, where carrying a
  # nil forward would fail somewhere else entirely.
  defp wait_for(check, tries \\ 60)
  defp wait_for(_check, 0), do: raise("the mesh daemon never became ready")

  defp wait_for(check, tries) do
    case check.() do
      falsy when falsy in [nil, false] ->
        Process.sleep(2_000)
        wait_for(check, tries - 1)

      value ->
        value
    end
  end

  defp url, do: System.get_env("ZTL_CONTROLLER_URL")
  defp key, do: System.get_env("ZTL_AUTH_KEY")
end

The complete sequence is verified with nerves_system_rpi4 and LOLIPOP ZTL. Tailscale and NetBird use the same Nerves facilities, but their daemon flags and startup behavior still need validation on the target system before they are used in a firmware release.

Usage

Every node must run with a longname built from its mesh IPv4 address and a shared basename. The simplest distribution setup is EPMD-less with a fixed port, one BEAM node per host:

elixir --name myapp@100.64.0.7 --cookie "$CLUSTER_COOKIE" \
  --erl "-start_epmd false -erl_epmd_port 45892" \
  -S mix run --no-halt

-erl_epmd_port doubles as the local listen port and the port assumed for every remote node, so all nodes must use the same one. Distribution traffic is plaintext, but it only flows inside the WireGuard-encrypted mesh. The listener binds all interfaces by default; to keep it mesh-only add -kernel inet_dist_use_interface '{100,64,0,7}'.

Tailscale

config :libcluster,
  topologies: [
    tailnet: [
      strategy: Cluster.Strategy.TailscaleLocal,
      config: [
        node_basename: "myapp",
        hostnames: ["myapp-"]
      ]
    ]
  ]

NetBird

config :libcluster,
  topologies: [
    netbird: [
      strategy: Cluster.Strategy.Netbird,
      config: [
        node_basename: "myapp",
        hostnames: ["myapp-"]
      ]
    ]
  ]

LOLIPOP Zero Trust Link

Run ztld in TUN mode (in userspace mode there is no TUN device, so the BEAM cannot bind its distribution listener to the mesh address).

config :libcluster,
  topologies: [
    ztl: [
      strategy: Cluster.Strategy.LolipopZTL,
      config: [
        node_basename: "myapp",
        hostnames: ["myapp-"]
      ]
    ]
  ]

WireGuard

Plain WireGuard has no hostnames, so members are selected by an IPv4 cidr over the peers' allowed-ips, and a peer counts as online when its latest handshake is recent (handshake_max_age, default 180s). Keep persistent_keepalive enabled on the mesh so idle peers stay visibly alive.

config :libcluster,
  topologies: [
    wireguard: [
      strategy: Cluster.Strategy.WireGuard,
      config: [
        node_basename: "myapp",
        interface: "wg0",
        cidr: "10.77.0.0/24"
      ]
    ]
  ]

Options

All strategies accept:

• node_basename (required) — node name part before @, shared by all cluster nodes. It is that part and nothing else, so an @ or whitespace in it is refused: the node name it would build is one no node answers to, and every connect would simply fail
• cli — path to the mesh CLI binary (defaults: "tailscale" / "netbird" / "ztlctl" / "wg")
• polling_interval — poll interval in ms (default 5_000)
• cmd_timeout — CLI timeout in ms (default 10_000)
• runner — a LibclusterMesh.Runner implementation, injectable for tests

polling_interval and cmd_timeout are millisecond counts handed to a BEAM timer, which takes at most 4_294_967_295 of them (a little under 50 days); a larger one is refused at startup rather than raising mid-poll.

Member selection per strategy:

• hostnames (required by TailscaleLocal / Netbird / LolipopZTL) — list of mesh hostname prefixes identifying cluster members; other mesh peers (laptops, phones, …) are ignored. Give your cluster hosts a common prefix (myapp-1, myapp-2, …)
• cidr + interface (required by WireGuard) — members are the peers on one named interface whose allowed-ips hold a /32 inside cidr, which is the address they are dialled at; "all" and "interfaces" are reserved wg show operands and are refused. A peer whose only entry covers a range is advertising a route rather than a host, and is not a member. handshake_max_age (default 180 seconds) decides how fresh a handshake must be to count as online

Cluster.Strategy.LolipopZTL additionally accepts socket — the ztld socket path passed to ztlctl --socket. Without it, ztlctl selects the daemon socket.

A key a strategy does not support raises ArgumentError at startup rather than being ignored — a cidr: handed to Tailscale, a socket: handed to NetBird, or a mistyped hostnames, otherwise means no peer ever matches or no setting ever takes effect, so the cluster never forms and nothing says why.

Relation to libcluster_tailscale

libcluster_tailscale (Cluster.Strategy.Tailscale) solves the same problem with a different information source: it queries the Tailscale cloud API (api.tailscale.com, authenticated with a tskey-api-… key), while this library asks the node's own daemon. Neither replaces the other; pick by constraint:

Having an API key does not conflict with this library — TailscaleLocal simply never needs it. The module is named TailscaleLocal (after tailscaled's own "LocalAPI") precisely so both libraries can coexist in one project without a module clash, e.g. daemon-run nodes discovered locally and daemon-less nodes discovered via the API:

config :libcluster,
  topologies: [
    tailnet: [strategy: Cluster.Strategy.TailscaleLocal, config: [...]],
    via_api: [strategy: Cluster.Strategy.Tailscale, config: [authkey: "tskey-api-...", ...]]
  ]

How it works

Each poll runs the mesh CLI through LibclusterMesh.Runner (over a port, with a timeout covering the whole call and every failure returned rather than raised, so a missing binary or a hung daemon can never take the strategy down), parses the peer list into LibclusterMesh.Peer structs, keeps the online peers that belong to the cluster — by hostname prefix on the meshes that have hostnames, by IPv4 CIDR on plain WireGuard, which does not — maps them to basename@mesh_ip node names, and hands them to Cluster.Strategy.connect_nodes/4. The next poll is scheduled only after the current one finishes, so polls never overlap. Nodes joining the mesh are picked up on the next poll. The strategy does not disconnect a node merely because a later daemon response omits it or marks it offline: a mesh control-plane interruption can produce that view while the direct data path and Erlang connection remain healthy. The mesh removes a lost route, and :net_kernel closes the unusable connection after net_ticktime.

Two consequences of that design are worth knowing before you rely on the poll interval:

• An unreachable member delays the next poll.Cluster.Strategy.connect_nodes/4 works through the nodes to connect one at a time, and :net_kernel.connect_node/1 blocks for several seconds against a host that does not answer. Since the next poll is only scheduled once the current one returns, N unreachable members push it out by N × that wait. Nothing is lost — discovery simply runs slower than the configured interval until those members answer or leave the mesh — but a mesh with many dead entries is a mesh whose discovery is slow.
• The WireGuard strategy only sees one interface's peers. It reads wg show <interface> dump, which lists the peers configured on that interface and nothing else. In a hub-and-spoke setup a spoke has only the hub as a peer, so it discovers the hub and never the other spokes; a full cluster needs every node configured with every other as a WireGuard peer. The three named meshes ask a daemon that knows the whole network, so this applies to Cluster.Strategy.WireGuard alone.

The test suite covers captured daemon response shapes, command failures, configuration boundaries, member selection, and the complete path from each strategy's runner through Cluster.Strategy.connect_nodes/4. Network-level validation still belongs to the deployment environment because loopback cannot reproduce latency, packet loss, or asymmetric reachability.

Development

mix test          # unit tests and doctests
mix check         # format check, warnings-as-errors compile, tests, credo, dialyzer

License

MIT