Transmuter ⚗️

Transmuter runs Hugging Face ONNX models on any Nx backend, with no per-architecture code. Use EMLX on Apple silicon, or EXLA on CUDA or on the CPU.

The goal of Transmuter is to implement the ONNX operators in Nx. Then ONNX models run without model code.

An ONNX file contains a graph of operators. An operator is one small, standard computation, such as MatMul, Conv, Softmax, Add or Reshape. Each operator has inputs, outputs and attributes, for example the kernel size of a Conv. The ONNX standard defines about 200 operators, and a model connects a few hundred to a few thousand of them. When a model is exported, the exporter traces its forward pass and writes these operators to the file.

Transmuter implements each operator one time, as Nx operations. It then runs a model by running its graph, operator by operator. So Transmuter does not port model architectures. A new architecture needs no new Elixir code. It needs only operators that Transmuter already has.

The gap

Elixir has three ways to run Hugging Face models today. Each way has a limit:

• Bumblebee runs models with great accuracy on all Nx backends. But a person must port each architecture to Axon by hand, and a port takes weeks of work and review. Hugging Face gets new architectures faster than people can port them.
• Ortex runs ONNX models with ONNX Runtime. But it runs outside of Nx: you cannot put it in a defn function, it has no Apple silicon GPU path, and it is in maintenance mode.
• axon_onnx changed ONNX graphs into Axon models. It stopped in 2024. It set all shapes when it loaded a model, and many ONNX graphs need shapes that change.

Transmuter fills this gap. It runs the exported graph directly as Nx operations, so:

• Models that Bumblebee does not have run with no new code. Examples are SAM, Depth Anything, Grounding DINO, OWLv2, Florence-2, RF-DETR, Kokoro and Stable Diffusion.
• The result is plain Nx. It runs on EMLX and EXLA, it works with Nx.Serving, and Nx.Defn.grad can differentiate it.
• Shapes are set when the function compiles, not when the model loads. Thus decoders with a KV cache and graphs with If branches work.

When a model has no ONNX file on Hugging Face, export it one time with Optimum. In the 300 most downloaded transformers models, 62% of downloads already have an ONNX file on Hugging Face.

Example

Transmuter.Text gives servings that take text, as in Bumblebee. A serving loads the ONNX file, the tokenizer and the configuration of a Hugging Face repository, and returns an Nx.Serving:

opts = [backend: EMLX.Backend, defn_options: [compiler: EMLX]]

serving = Transmuter.Text.text_embedding({:hf, "Xenova/all-MiniLM-L6-v2"}, opts)
Nx.Serving.run(serving, "The cat sat on the mat.")
#=> %{embedding: #Nx.Tensor<f32[384] ...>}

serving = Transmuter.Text.generation({:hf, "onnx-community/Qwen3-0.6B-ONNX"}, opts)
Nx.Serving.run(serving, %{messages: [%{role: "user", content: "What is the capital of France?"}], seed: 7})
#=> %{results: [%{text: "<think>\n...</think>\n\nThe capital of France is **Paris, France**.",
#=>               token_summary: %{input: 15, output: 167, padding: 0}}]}

The tasks:

• Transmuter.Text
  • text_embedding: MiniLM, BGE, EmbeddingGemma and Qwen3-Embedding
  • text_classification: DistilBERT and other sequence classifiers
  • text_ranking: MS MARCO cross-encoders and Qwen3-Reranker
  • zero_shot_entity_recognition: GLiNER (span and token models)
  • zero_shot_classification: DistilBERT MNLI and other NLI models
  • token_classification: BERT NER
  • fill_mask: DistilRoBERTa
  • question_answering: DistilBERT SQuAD
  • generation: SmolLM2, SmolLM3, Qwen3 and Gemma 3
  • text_to_text: T5 and Marian, and NLLB and M2M100 for translation between many languages
• Transmuter.Vision
  • image_classification: ViT
  • image_embedding: CLIP, SigLIP and SigLIP2
  • zero_shot_image_classification: CLIP, SigLIP and SigLIP2
  • image_to_text: ViT-GPT2, TrOCR and Florence-2
  • image_text_to_text: chats about images with SmolVLM, Qwen2-VL, Qwen2.5-VL, Qwen3-VL, Ministral 3, Gemma 3 and LLaVA
  • object_detection: DETR, YOLOS, RT-DETR and RF-DETR
  • zero_shot_object_detection: OWL-ViT, OWLv2 and Grounding DINO
  • depth_estimation: Depth Anything
  • image_segmentation: Segformer
  • background_removal: ORMBG
  • mask_generation: SAM, SAM 2 and the SAM 3 tracker, with point and box prompts
  • image_to_image: Swin2SR super resolution
• Transmuter.Audio
  • speech_to_text: Whisper
  • audio_classification: AST and wav2vec2
  • zero_shot_audio_classification: CLAP
  • text_to_speech: VITS and SpeechT5

The results have the shapes of Bumblebee or of Python transformers, and their values match the pipelines of Python transformers. The text of image_text_to_text matches ONNX Runtime on the same files. See the Pipelines guide.

Tensors

For a model that has no serving, give tensors to the graph:

model = Transmuter.load!({:hf, "Xenova/all-MiniLM-L6-v2", "onnx/model.onnx"}, backend: EMLX.Backend)

inputs = %{
  "input_ids" => Nx.tensor([[101, 7592, 2088, 102]]),
  "attention_mask" => Nx.tensor([[1, 1, 1, 1]]),
  "token_type_ids" => Nx.tensor([[0, 0, 0, 0]])
}

run = Transmuter.compile(model, inputs, compiler: EMLX)
%{"last_hidden_state" => hidden} = run.(inputs)

# Sentence embedding: mean of the token vectors, then L2 normalization.
mask = Nx.new_axis(inputs["attention_mask"], -1)
embedding = Nx.sum(Nx.multiply(hidden, mask), axes: [1]) |> Nx.divide(Nx.sum(mask, axes: [1]))
embedding = Nx.divide(embedding, Nx.LinAlg.norm(embedding, axes: [1], keep_axes: true))

The export sets the names of the inputs and outputs. model.inputs and model.outputs show them. For this export, model.outputs is ["last_hidden_state"].

Transmuter runs the graph that is in the file, and nothing more. The graph of all-MiniLM-L6-v2 stops at the token vectors, because sentence-transformers adds the pooling outside of the model. Thus the example above does the pooling. Transmuter.Text.text_embedding/2 does the tokenization, the pooling and the batching for you.

Transmuter.compile/3 compiles the model for the shapes of the inputs that you give. The function that it returns also takes inputs of other shapes: it compiles them on their first call, and keeps the most recently used shapes compiled. The :cache_size option sets how many, and the default is 8.

Transmuter.run/4 is the function that Transmuter.compile/3 compiles. It reads the ONNX graph and calls one Nx function for each operator. You can use it in two ways:

• Call it directly. Each operator runs immediately, one after the other. This is eager execution. It is slower, but the model can read values while it runs.
• Give it to Nx.Defn.jit/2 or Nx.Defn.value_and_grad/2. Nx then records the operators as one graph, and the backend compiles that graph.

Installation

def deps do
  [
    {:transmuter, "~> 1.0"},
    {:emlx, "~> 0.5"},
    {:tokenizers, "~> 0.5"}
  ]
end

Use EXLA instead of EMLX for CUDA. The Transmuter.Text servings need tokenizers. To read image files for the Transmuter.Vision servings, add {:stb_image, "~> 1.0"}. Chat templates need {:otera, "~> 0.1"}, which renders Jinja templates in pure Elixir.

Compilation and graph breaks

Some graphs use a tensor value to set a shape or to select a branch. Transmuter.compile/3 breaks the graph at these operators, runs the break eagerly on the real values, and compiles the next segment for the shapes the values decide. Compiled segments are cached by input shapes.

See the Compiling and Graph Breaks guide for a full explanation of graph breaks, static inputs, the :cache_size option, and how to read error messages. The Backends and Performance guide explains EMLX memory management.

See the graph

Transmuter.Graph shows the graph of a model with what Transmuter does to it. The fill of a node gives its segment, and a red node is a graph break with its reason. A node with static outputs has a dashed border, and a node that replaces fused nodes has a thick border. Click a node to see its details, such as the shapes of its outputs.

There are three ways to get the view:

• In Livebook, call Transmuter.Graph.as_graph(model, inputs: inputs). It needs the optional kino dependency.
• Outside Livebook, call Transmuter.Graph.to_html(model, "graph.html", inputs: inputs) and open the file in a browser.
• From the command line, run mix transmuter.graph onnx-community/Kokoro-82M-v1.0-ONNX --run --open. It calls the model once with inputs of ones.

The view loads no file from the network. For the graph as the exporter wrote it, without fusion and without a call, run mix transmuter.graph onnx-community/Kokoro-82M-v1.0-ONNX --plain.

Non-tensor values and random operators

Nx tensors hold only numbers. Transmuter keeps string tensors, sequences, and empty optionals as Elixir terms. Give string and sequence inputs as :static_inputs. A compiled function returns only tensors. To read string or sequence outputs, call Transmuter.run/4 without compilation. Random operators draw from Nx.Random, and each call of a compiled model draws new values. ImageDecoder needs the optional stb_image dependency.

See Supported Operators and Data Types for details.

Telemetry

Transmuter emits :telemetry events when it loads, compiles and runs a model. The graph break events show which segment compiles again and for which shapes, which segment the cache evicts, and how long each break operator runs. Transmuter.Telemetry lists each event with its measurements and metadata.

To log the compile work, attach the logger when your application starts:

Transmuter.Telemetry.Logger.attach()

Precision and quantization

Half-precision exports run in native f16 or bf16 by default, with half the memory of f32. compute_type: :f32 runs them in f32, as ONNX Runtime on the CPU does, for deep graphs that drift in native f16. With backend: EMLX.Backend, 4-bit and 8-bit MatMulNBits weights stay packed as MLX quantized tensors for lower peak memory and faster inference. On other backends, such as EXLA, these weights become float weights when the model loads, about 4 bytes per weight, so each call runs one matmul and does not unpack them.

See the Backends and Performance guide and the Language Models guide for details and benchmarks.

Fine-tuning

Transmuter.run/4 is plain Nx. Thus Nx.Defn.value_and_grad/2 calculates gradients through the graph, and an optimizer such as Polaris updates the params. Load the model with trainable: true. Then small weights, such as a bias with 1 element, are params and not constants.

bench/finetune.exs fine-tunes a MiniLM cross-encoder. Its gradients and its loss curve for 30 AdamW steps are the same as in PyTorch.

Fine-tune an fp32 export. Inference exports have no dropout, and quantized weights have no gradients.

Coverage

bench/coverage runs the most downloaded ONNX exports for each Hugging Face task. It compares each output with ONNX Runtime. The reference is ONNX Runtime with graph optimizations off, because its default optimizer gives wrong results for some exports. For MatMulNBits, the reference uses accuracy_level 0. At level 4, ONNX Runtime quantizes the activations to int8, and that result is an approximation.

Transmuter supports all 203 operators of the ONNX standard domain, and the ONNX Runtime contrib operators that Hugging Face exports use, including the mixture-of-experts operators MoE and QMoE.

These models do not match value by value. Most of them make small rounding differences large:

• Grounding DINO selects 900 of 13,294 queries with TopK, and some scores differ by only 9.5e-7. All operators before TopK match. On test/fixtures/cats.png with three labels, Transmuter gives the same labels and number of detections as ONNX Runtime, with scores within 0.001.
• The Gemma 4 E2B decoder has activations up to about 1300 and final norm weights up to about 700. When the test changes the float inputs of ONNX Runtime by 1e-7 of their values, about one f32 rounding step, the logits of ONNX Runtime change by up to 5.5. The logits of Transmuter differ from ONNX Runtime by 5.3 on the CPU and by 13.6 on the GPU, which is the same noise. The two give the same token for 7 of 8 positions of the random test input. That input has random embeddings with sigma 0.5, without the scale of the real embeddings. With a real prompt of 20 tokens through embed_tokens_q4 and decoder_model_merged_q4, Transmuter on the GPU gives the 32 greedy tokens of ONNX Runtime on the CPU, with logits within about 1e-4 at each step, also at a step where the top two logits of ONNX Runtime are only 0.055 apart.
• Kokoro sums a phase over all audio samples and takes its STFT. ONNX Runtime and MLX round that sum differently. Also, the atan2 of the export gives a different phase for an exact STFT than for the STFT of ONNX Runtime (see The atan2 fix). When the test gives the Sin and STFT outputs of ONNX Runtime to Transmuter, all other operators match. With the atan2 fix of the example, the spectrum of the speech matches PyTorch, with no tone at 4.8 kHz.
• The int8 export of falcon-rw-1b uses DynamicQuantizeLinear and MatMulInteger. The ONNX Runtime reference ran on x86 with AVX2 and no VNNI. There, the MatMulInteger kernel of ONNX Runtime adds each pair of u8 × s8 products in int16, and a pair above 32767 saturates. Transmuter gives the exact int32 sum, which is the result that the operator defines. Layers 0 to 2 match, and one of the 6144 values of the layer 3 query_key_value product differs by 912. A numpy product with the int16 saturation gives the output of ONNX Runtime exactly, and the exact product gives that of Transmuter. The logits then differ by up to 1.46. ONNX Runtime on arm64 (macOS) gives the result of Transmuter, with logits within 1.9e-5.
• The int8 export of slovak-gpt-j-405M differs from ONNX Runtime on the same machine, with logits up to 0.83 apart, for two reasons. First, 9 of its 145 MatMulInteger nodes, in the If branch of the merged decoder, saturate on x86, from the layer 0 k_proj on. For each of them, a numpy product with the int16 saturation gives the output of ONNX Runtime exactly, and Transmuter gives the exact product. Second, on arm64, where all 145 products are exact, one DynamicQuantizeLinear value of layer 8 rounds to the other integer. Its x / scale is -1.4999986, only 1.4e-6 from a .5 tie, so the float differences of about 1e-6 between two machines decide the rounding. That one quantum grows to logits 0.74 apart. This is rounding noise, not a bug.

These models also ran from input to final result, and the result agrees with ONNX Runtime:

The Florence-2 vision encoder

The vision_encoder.onnx of onnx-community/Florence-2-base-ft gives other image features than PyTorch. Transmuter gives the features of ONNX Runtime, so its text is that of the export. The channel attention of DaViT multiplies the queries by tokens ** -0.5. The export computed this scale for a 224x224 input and stored it as a constant: the 12 Mul nodes /blocks.N/blocks.N.M/channel_block/channel_attn/fn/Mul have 1/56 (stage 0), 1/28 (stage 1), 1/14 (the 9 blocks of stage 2) and 1/7 (stage 3). The image processor resizes each image to 768x768, so the correct values are 1/192, 1/96, 1/48 and 1/24. The softmax of each channel attention is thus 3.43 times too sharp. The export of Xenova/tiny-random-Florence2ForConditionalGeneration has the same constants. On test/fixtures/cats.png, the image features differ from PyTorch by up to 6.1 (mean 0.44). With the correct constants, they differ by up to 1.9e-4.

The text changes for all 6 tasks of the tests (beam search with 3 beams):

With the correct constants, the export gives the text of PyTorch for all 6 tasks. To get it, change the 12 initializers of the Mul nodes to tokens ** -0.5 for a 768x768 input, as test/support/pipeline_references.py does, and save the file.

To run the matrix:

1. Run bench/coverage/select_models.py to select the models.
2. Run bench/coverage/record_references.py to download them and record the ONNX Runtime outputs.
3. Run bench/coverage/run_cases.py to run each case through Transmuter.
4. Run bench/coverage/print_matrix.py to show the matrix, and bench/coverage/list_models.py to make the table of tested models.

If a case does not match, run bench/coverage/trace_ort.py and bench/coverage/find_divergence.exs. They show the first node where Transmuter and ONNX Runtime give different values.

Tests

mix test downloads five models and compares them with ONNX Runtime. test/support/model_references.py shows the reference values.

mix test also runs one small graph for each operator variant in test/fixtures/ops, with no download. test/support/make_op_fixtures.py makes these graphs and records their outputs from ONNX Runtime, or from the ONNX reference evaluator when ONNX Runtime does not have the operator. For random operators, the tests check the shape, the type and the distribution of the values.

On a Mac, TRANSMUTER_TEST_DEVICE=cpu mix test uses the EMLX CPU device. But EMLX 0.5 runs compiled functions on the GPU when Metal is available, also with device: :cpu. Thus only the Linux CI tests the CPU fully.

License

Apache-2.0