MLServe

Production machine-learning inference for the BEAM.
OTP supervision, worker pools, dynamic batching, caching, telemetry and zero-downtime model rollout — around any ML backend.

Elixir has excellent model libraries — Nx, Bumblebee, Ortex, EXLA. What it has not had is the boring layer around them: the thing that supervises a model, pools access to it, batches requests, caches results, reports latency, and lets you swap a model version without a deploy.

Every team that puts a model into a Phoenix app rebuilds that layer by hand. MLServe is that layer, and nothing else. It does not train, it does not own a tensor type, and it is not another LLM API wrapper.

Works with any Elixir ML backend — Nx and Nx.Serving, Bumblebee, ONNX Runtime via Ortex, scikit-learn or PyTorch exports, a Python process over a port, or a remote HTTP model server. If it can be reached from a function, MLServe can supervise, pool, batch, cache, measure and version it.

MLServe.predict(:fraud_detection, %{
  amount: 1500.50,
  transaction_count_24h: 8,
  failed_transactions_24h: 2
})
#=> {:ok, %{prediction: :fraud, probability: 0.94}}

Installation

Add ml_serve to your dependencies in mix.exs:

def deps do
  [{:ml_serve, "~> 0.1.0"}]
end

mix deps.get

Requires Elixir 1.14 or later. The only runtime dependency is :telemetry.

Why it exists

Inference serving is a concurrency problem wearing a machine-learning hat: parallel requests, stateful and expensive-to-load models, backends that are not thread-safe, failures that must stay contained, and versions that must change without downtime. That list is OTP's home ground.

What that buys you concretely:

Per-model isolation. Each loaded model version is its own supervision subtree. A backend that crash-loops exhausts its own restart budget and marks itself failed. Other models keep serving.
No dispatch overhead. Finding a model is one lock-free ETS read in the calling process. MLServe never puts a GenServer between your request and the model.
Shared-state backends cost nothing. A backend safe to call concurrently — an Nx.Serving, a pure function, a remote service — runs in the caller with state read from :persistent_term. No worker processes, and your tensors are never copied between mailboxes.
Dynamic batching. Independent concurrent callers are coalesced into one backend call, which is the difference between a busy GPU and an idle one.
Zero-downtime model upgrades. Load a new version beside the running one, send it 5% of traffic, compare per-version telemetry, promote, drain the old one.

Quick start

Any module with load/1 and predict/2 is a model:

defmodule MyApp.FraudModel do
  @behaviour MLServe.Model

  @impl true
  def load(config), do: {:ok, Keyword.fetch!(config, :threshold)}

  @impl true
  def predict(threshold, %{amount: amount}) do
    probability = min(amount / 2000, 1.0)

    {:ok, %{
      prediction: if(probability > threshold, do: :fraud, else: :legitimate),
      probability: probability
    }}
  end
end

config :ml_serve,
  models: [
    fraud_detection: [
      backend: MyApp.FraudModel,
      version: "1.0.0",
      workers: 4,
      config: [threshold: 0.7]
    ]
  ]

…or at runtime, which is the same code path:

MLServe.load_model(:fraud_detection, backend: MyApp.FraudModel, config: [threshold: 0.7])

Then predict from anywhere — a controller, an Oban job, a Task:

case MLServe.predict(:fraud_detection, %{amount: 1500.50}) do
  {:ok, result} -> result
  {:error, reason} -> Logger.warning("inference failed: #{inspect(reason)}")
end

Architecture

MLServe.Supervisor                     (:rest_for_one)
├── MLServe.Registry                   process registry, partitioned by scheduler
├── MLServe.ModelRegistry              owns the catalog ETS table
├── MLServe.Cache                      owns the cache ETS table + TTL sweeper
├── MLServe.TaskSupervisor             batch fan-out
└── MLServe.ModelSupervisor            DynamicSupervisor
      └── MLServe.ModelInstance        one subtree per loaded {name, version}
            ├── MLServe.ModelServer    lifecycle, status, graceful drain
            ├── MLServe.WorkerSupervisor
            │     └── MLServe.Worker × N
            └── MLServe.Batcher        when dynamic batching is configured

:rest_for_one at the top is deliberate: MLServe.ModelRegistry owns the catalog ETS table, and ETS tables die with their owner. If it restarts, every route has evaporated and any model still running would be serving traffic the registry no longer knows about.

The prediction path itself contains no MLServe process at all:

route lookup (1 ETS read) → admission control (atomic counter) → telemetry span opens
  → cache lookup → preprocess hook → dispatch → postprocess hook → cache write

Cache, hooks and validation all run before dispatch, so a worker is occupied only for actual inference. A preprocess hook that queries Postgres for stored features runs on the caller's own scheduler time, never on a GPU worker's.

See the Architecture guide for the full picture.

Concurrency

A backend declares how it may be executed, and MLServe picks a completely different strategy:

`concurrency`	Where `predict/2` runs	Where state lives	Use for
`:shared`	The calling process — no processes, no copies	`:persistent_term`	`Nx.Serving`, Bumblebee, pure functions, remote services
`:exclusive`	A pooled worker	Worker state, or a shared handle	ONNX Runtime sessions, Python ports, anything not thread-safe

Crossed with when the model loads:

`load`	Meaning
`:once` (default)	`load/1` runs once; the state term is shared by all workers. Correct for NIF-resource models, where the term is a cheap handle.
`:per_worker`	`load/1` runs per worker. Correct for ports and per-worker sessions.

:once as the default matters — loading a 2 GB model separately into eight workers is an out-of-memory crash, not a pool.

models: [fraud_detection: [workers: 8, selection: :least_loaded, max_concurrency: 64]]

Model lifecycle

MLServe.load_model(:fraud, backend: MyApp.Model, version: "2.1.0")
MLServe.await_ready({:fraud, "2.1.0"})   # loading is async; boot is never blocked

MLServe.canary(:fraud, "2.1.0", 5)       # 5% of traffic, tagged in telemetry
MLServe.promote(:fraud, "2.1.0")         # atomic pointer flip, no restart
MLServe.unload_model(:fraud, version: "1.0.0")   # drains in-flight requests first

MLServe.model_status(:fraud)
#=> {:ok, %{status: :ready, workers: 4, in_flight: 3, requests: 154_223, errors: 12, ...}}

Loading is asynchronous and retried with backoff, so a model on a network mount that attaches a moment after the container starts does not take your application down with it. MLServe.ready?/1 is your readiness probe.

Batch inference

MLServe.batch_predict(:fraud_detection, [features_a, features_b, features_c])

One backend call when the backend implements batch_predict/2, a mapped predict/2 otherwise.

For many concurrent callers rather than one caller with many inputs, enable dynamic batching:

models: [fraud_detection: [batching: [max_size: 16, timeout: 10]]]

Independent predict/3 calls arriving within the window are coalesced into a single backend invocation. Watch [:ml_serve, :batch, :flush] — a healthy setup flushes mostly on :full.

Telemetry

[:ml_serve, :prediction, :start | :stop | :exception]
[:ml_serve, :model, :load]  [:ml_serve, :model, :unload]
[:ml_serve, :cache, :hit]   [:ml_serve, :cache, :miss]
[:ml_serve, :batch, :flush]

Measurements include duration, queue_duration, inference_duration and batch_size; metadata carries model, version, backend, cached? and canary?.

MLServe.Telemetry.Logger.attach(level: :info)          # zero-dependency visibility
MLServe.Telemetry.Metrics.metrics()                    # Telemetry.Metrics / LiveDashboard

Because version and canary? ride along on every event, comparing a canary against the incumbent needs no extra instrumentation — which is what makes a promote/rollback decision possible.

Backends

MLServe ships two dependency-free backends: MLServe.Backend.Function (wrap any function or MFA) and MLServe.Backend.Static (fixed result — swap it in config/test.exs to test your app without a model).

Real ML runtimes are guides with complete implementations, not dependencies, so the core stays at one runtime dep and backends version independently:

Phoenix integration

Phoenix is not a dependency. It does not need to be:

defmodule MyAppWeb.MLController do
  use MyAppWeb, :controller

  def predict(conn, params) do
    case MLServe.predict(:fraud_detection, params) do
      {:ok, result} -> json(conn, result)
      {:error, reason} -> conn |> put_status(status_for(reason)) |> json(%{error: inspect(reason)})
    end
  end
end

See the Phoenix guide for status-code mapping, LiveDashboard metrics and readiness probes, and the Oban guide for asynchronous inference with retry semantics driven by MLServe.Error.retryable?/1.

Errors

Every function returns {:ok, result} or {:error, reason}; bang variants raise MLServe.Error.

Reason	Meaning
`:model_not_found`	Not registered under that name or version
`:model_not_ready`	Registered, but loading, draining or failed
`:timeout`	The deadline passed before inference completed
`:overloaded`	At the `:max_concurrency` limit
`{:invalid_input, reason}`	Rejected by a `:preprocess` hook
`{:batch_too_large, max}`	Batch exceeded `:max_batch_size`
`{:backend_error, %MLServe.BackendError{}}`	The backend raised — exception and stacktrace preserved
`{:load_failed, reason}`	The model could not be loaded

Backend failures are never swallowed: a raise is caught only to attach the model, version, callback and stacktrace before surfacing it.

Security

Model paths are validated against a configured :model_root, with .. traversal and symlink escape rejected, plus existence, readability and size checks.
Optional checksum: {:sha256, "..."} verification on load.
Backend modules are verified to implement MLServe.Model at load time, not assumed.
MLServe never calls binary_to_term/1, Code.eval_*, or loads a NIF from a model artifact. A model file is data. Supplying one is not a way to execute code.

Common questions

How do I serve an ONNX model in Elixir?

Wrap an Ortex session in a MLServe.Model backend and MLServe supplies the pool, batching, checksum verification and telemetry around it. There is a complete runnable example in examples/onnx — including HuggingFace all-MiniLM-L6-v2 doing real semantic search — and a full implementation in Creating a Model Backend.

How do I run Bumblebee or `Nx.Serving` in production Phoenix?

Declare the backend concurrency: :shared. Inference then runs in the Phoenix request process with state read from :persistent_term — no worker pool, no message copies of your tensors, and no serialisation point. See Concurrency and Phoenix Integration.

Is this an Elixir alternative to TensorFlow Serving or TorchServe?

For the serving concerns, yes — supervision, pooling, dynamic batching, caching, metrics, versioning and canary rollout — but in-process on the BEAM instead of as a separate service you deploy, scale and monitor. You keep one deployable and lose a network hop. It does not bring its own model runtime; you point it at Nx, ONNX Runtime, or whatever you already use.

Can I deploy a new model version without a deploy?

Yes — that is the point of MLServe.load_model/2, MLServe.canary/3 and MLServe.promote/2. Load the new version beside the running one, send it a percentage of traffic, compare per-version telemetry, then promote with a single ETS write. In-flight requests finish on the version they started on. See Model Versioning.

How do I test application code that calls a model?

Point the model at MLServe.Backend.Static in config/test.exs. Your suite then exercises the real routing, caching, telemetry and error handling with no model file, no ML runtime and no mocking library — see examples/inference_service, whose tests do exactly this.

What does it cost on the hot path?

One lock-free ETS read in the calling process. MLServe.ModelRegistry owns the catalog but is never in the request path, and the only MLServe process involved in a prediction is the worker running inference — none at all for :shared backends. The single runtime dependency is :telemetry.

Documentation

Full documentation is on HexDocs, including guides for Getting Started, Architecture, Creating a Model Backend, Running Inference, Batch Inference, Concurrency, Telemetry, Model Versioning, Phoenix, Oban and Production Deployment.

Examples

examples/ has runnable code for everything above — it lives on GitHub rather than in the Hex package:

Seven scripts, one command each, covering concurrency, batching, caching, versioning, telemetry and the full error taxonomy: elixir examples/scripts/01_quick_start.exs
Three Livebook notebooks for the same ground, interactively.
An HTTP inference service on Bandit and Plug — models declared in configuration, both execution strategies, and a full mapping from MLServe's error taxonomy onto HTTP status codes.
Three ONNX models loaded through ONNX Runtime via Ortex: a convenient export covering :model_root path containment, :checksum enforcement and one session shared across a worker pool; a PyTorch-exported GPT-NeoX with a batch dimension pinned to 1, showing what a backend must declare when the model cannot do what you would like; and all-MiniLM-L6-v2 from HuggingFace doing real semantic search, with tokenization inside the backend and the model fetched and checksum-verified at first run rather than committed.

For AI assistants and LLM tooling

usage-rules.md is a condensed, machine-readable summary of the public API and its constraints — including the mistakes that are easy to make — and llms.txt points at the full documentation set. Point your coding agent at either; both ship inside the Hex package.

Contributing

Issues and pull requests are welcome. Before submitting:

mix lint    # format --check-formatted + compile --warnings-as-errors + credo --strict
mix test
mix dialyzer