RpcLoadBalancer

An Elixir library for executing Remote Procedure Calls across distributed BEAM nodes with a built-in load balancer. It wraps Erlang's :erpc module with structured error handling and provides a pluggable node selection layer powered by OTP's :pg process groups.

Why This Exists

OTP's built-in node connection list (Node.list/0) does not automatically remove nodes that have crashed or become unreachable — they linger until the net kernel detects the failure, which can take seconds or longer depending on heartbeat configuration. During that window, any RPC call routed to the stale node will hang until it times out.

This library solves the problem by using :pg process groups instead of the raw node list. When a node goes down, its process group members are removed immediately because the backing processes exit. The load balancer only ever selects from nodes that have a live, registered process, so stale entries are never returned.

This gives you:

• Instant removal — dead nodes disappear from the selection pool as soon as their processes exit, with no timeout window
• Accurate membership — the node list always reflects actually reachable nodes
• Structured errors — instead of silent timeouts, callers get {:error, %ErrorMessage{code: :service_unavailable}} when no nodes are available

Features

• RPC wrappers — call/5 and cast/5 around :erpc with ErrorMessage error tuples
• Distributed load balancer — automatic node discovery and registration via :pg
• Seven selection algorithms — Random, Round Robin, Least Connections, Power of Two, Hash Ring, Weighted Round Robin, Call Direct
• Custom algorithms — implement the SelectionAlgorithm behaviour to add your own
• Node filtering — restrict which nodes join a balancer with string or regex patterns
• Connection tracking — atomic counters for connection-aware algorithms
• Random-node helpers — call_on_random_node/5 and cast_on_random_node/5 for name-based node filtering with built-in retry
• Graceful draining — in-flight call tracking and connection draining on shutdown
• Configurable retry — automatic retry with configurable count and sleep when no nodes are available

Installation

Add rpc_load_balancer to your dependencies:

def deps do
  [
    {:rpc_load_balancer, "~> 0.1.0"}
  ]
end

Quick Start

Direct RPC

{:ok, result} =
  RpcLoadBalancer.call(
    :"worker@host",
    MyModule,
    :some_fun,
    ["arg"],
    timeout: :timer.seconds(5)
  )

:ok = RpcLoadBalancer.cast(:"worker@host", MyModule, :some_fun, ["arg"])

Load-Balanced RPC

Start a load balancer, then route calls through it with the :load_balancer option:

{:ok, _pid} =
  RpcLoadBalancer.start_link(
    name: :my_balancer,
    selection_algorithm: RpcLoadBalancer.LoadBalancer.SelectionAlgorithm.RoundRobin
  )

{:ok, result} =
  RpcLoadBalancer.call(node(), MyModule, :my_fun, [arg], load_balancer: :my_balancer)

When the :load_balancer option is present, the first argument (node) is ignored — the balancer selects the target node for you.

Supervision tree ordering: The load balancer should be the last child in your supervision tree. OTP shuts down children in reverse order, so placing it last means it shuts down first during deployment — the node deregisters from the :pg group and drains in-flight calls before your application logic stops.

children = [
  MyApp.Repo,
  MyApp.Endpoint,
  {RpcLoadBalancer,
   name: :my_balancer,
   selection_algorithm: RpcLoadBalancer.LoadBalancer.SelectionAlgorithm.RoundRobin}
]

Algorithms

Numbers from mix run bench/select_node_bench.exs on Apple M1 Max, Elixir 1.19.5 / OTP 28.3.3, 8-node synthetic cluster, single process. Full per-cluster-size numbers, parallel-contention results, and identified bottlenecks are in bench/README.md.

Picking an algorithm. Use Random or RoundRobin unless you have a reason not to — they're the cheapest and distribute load uniformly. Reach for PowerOfTwo over LeastConnections once the cluster is large enough that linear scanning matters (~8+ nodes); it gives near-identical distribution at a fraction of the cost. HashRing is the most expensive single-process choice and the worst under parallel read contention — only use it when you actually need key affinity.

Configuration

All values are optional and can be set via application config:

config :rpc_load_balancer,
  call_directly?: false,
  retry?: true,
  retry_count: 5

Testing

In tests you typically don't have a multi-node cluster. Use the CallDirect algorithm so the load balancer executes calls locally instead of through :erpc:

{:ok, _pid} =
  RpcLoadBalancer.start_link(
    name: :my_balancer,
    selection_algorithm: RpcLoadBalancer.LoadBalancer.SelectionAlgorithm.CallDirect
  )

{:ok, result} =
  RpcLoadBalancer.call(node(), MyModule, :my_fun, [arg], load_balancer: :my_balancer)

To switch automatically based on environment, use a compile-time module attribute:

@selection_algorithm if Mix.env() === :test,
                       do: RpcLoadBalancer.LoadBalancer.SelectionAlgorithm.CallDirect,
                       else: RpcLoadBalancer.LoadBalancer.SelectionAlgorithm.RoundRobin

See the Testing with CallDirect how-to guide for full examples.

Documentation

This project's documentation follows the Diátaxis framework:

• Tutorial: Getting Started — learn the library by building a load-balanced RPC setup step by step
• How-To Guides — solve specific problems like custom algorithms, node filtering, and hash-based routing
• Reference — complete API documentation for every module
• Explanation — understand the design decisions and internal architecture

License

MIT — see LICENSE for details.