Mutare

Hex.pm Hexdocs CI License

Mutare is a mutation testing system for Elixir that mutates the source you actually write, and compiles once.

Mutation testing measures whether your test suite actually constrains the behavior of your code: it deliberately breaks your source code one small change at a time, and each time your tests nevertheless still pass that is an indication of a gap in your test suite.

Other Elixir mutation testing libraries, such as Muex, compile your code once per mutant, which can be quite slow, even with incremental recompilation. Mutation testing is never fast, but mutare aims to be the fastest Elixir mutation library with its compile-once approach. Mutare compiles a metamutant, a version of your program embedding all possible mutants behind runtime switches, and then selects the active mutant per test run via an environment variable.

Installation

The easiest way to install mutare is the igniter installer, which adds mutare to your :dev/:test deps and auto-configures framework integrations:

mix igniter.install mutare

It inspects your dependencies and, for each framework it finds, adds the matching companion package and wires it into a generated .mutare.exs:

Detected dependency Package added Wired into
:plug / :bandit / :plug_cowboy / :phoenix mutare_plug :mutators — Mutare.Plug.all/0
:phoenix mutare_phoenix :mutators — Mutare.Phoenix.all/0; :extensions — Mutare.Phoenix
:phoenix_live_view mutare_phoenix_live_view :mutators — Mutare.Phoenix.LiveView.all/0
:ecto_sql / :phoenix_ecto / :ecto mutare_ecto :mutators — {Mutare.Ecto, repo: YourRepo}
:oban / :oban_pro mutare_oban :mutators — Mutare.Oban.all/0
:decimal mutare_decimal :mutators — Mutare.Decimal.all/0
:swoosh / :phoenix_swoosh mutare_swoosh :mutators — Mutare.Swoosh.all/0
:phoenix_swoosh mutare_phoenix_swoosh :mutators — Mutare.Phoenix.Swoosh.all/0
:gettext mutare_gettext :extensions — Mutare.Gettext

A mutator package extends the :mutators list; a non-mutating extension like mutare_gettext (which defines how the built-in mutators handle a library's compile-time syntax) joins the :extensions list; mutare_phoenix does both, since its front module also routes Phoenix's compile-time macros. The Ecto repo is detected automatically (pass --repo MyApp.Repo to override). If you already have a .mutare.exs, it is left untouched and the recommended keys are printed for you to merge in.

You can install igniter globally, with mix archive.install hex igniter_new, or add it to your project's mix.exs:

{:igniter, "~> 0.8", only: [:dev]},

Or add mutare by hand — though if you're using any macro-heavy libraries, like ecto or gettext, you'll need to add the relevant mutator or extension too.

# mix.exs

{:mutare, "~> 0.4", only: [:dev, :test], runtime: false}

# Optionally, also:

# {:mutare_ecto, "~> 0.1", only: [:dev, :test], runtime: false}

# {:mutare_decimal, "~> 0.1", only: [:dev, :test], runtime: false}

# {:mutare_gettext, "~> 0.1", only: [:dev, :test], runtime: false}

# {:mutare_plug, "~> 0.1", only: [:dev, :test], runtime: false}

# {:mutare_phoenix, "~> 0.1", only: [:dev, :test], runtime: false}

# {:mutare_phoenix_live_view, "~> 0.1", only: [:dev, :test], runtime: false}

# {:mutare_swoosh, "~> 0.1", only: [:dev, :test], runtime: false}

# {:mutare_phoenix_swoosh, "~> 0.1", only: [:dev, :test], runtime: false}

Then run mix mutare.

Agent skill

Mutare ships a skill for coding agents that drive it: which scope suits which job, how to run it within an agent harness's limits, and how to triage survivors. With usage_rules, add to your mix.exs project config

usage_rules: [skills: [package_skills: [:mutare]]]

and run mix usage_rules.sync; mix igniter.install mutare adds that config for you when the project already depends on usage_rules. Without usage_rules, copy deps/mutare/usage-rules/skills/mutare into your agent's skills directory (.claude/skills/ for Claude Code).

How it works

  1. Transform. Every in-scope source file is rewritten into a metamutant that embeds all of its mutants. Mutare transforms the code you write, before macro expansion—so any macros that don't accept arbitrary expressions will probably need their arguments routed :raw in your config (see "Routing calls" below).
  2. Compile once. The metamutant compiles a single time. The source code doesn't change between runs, so there is no per-mutant recompilation.
  3. Run the suite per mutant. A baseline test run must pass; a coverage probe then maps each mutant to the test files that exercise it. Each mutant runs in a fresh mix test OS process, :workers at a time, each capped by a wall-clock timeout. The active mutant is selected using MUTARE_MUTANT_NAMESPACE (its file) and MUTARE_ACTIVE_MUTANT (its id within that file).
  4. Report. Surviving mutants are listed in an abbreviated format as the run progresses, and you get a full report, with diffs and a mutation score, at the end. You can also enable JSON, HTML, or SARIF format output.

Nevertheless, note that a suite that is green under plain mix test can fail during Mutare's baseline run when a test asserts exact FunctionClauseError fields or stacktrace frames, because Mutare's function lifting renames those internals. See "Troubleshooting baseline-only failures" in the mix mutare task docs (mix help mutare) for the mechanics and the skip_lifting escape hatch.

Features

A known-equivalent mutant can be supressed with a comment. A trailing comment marks the line it is on as ignored, and a standalone comment applies to the next line. Ignored mutants are excluded from the score and their generated code is omitted from the metamutant. They retain their report entries and positions within --max-mutants.

def discounted(amount, percent), do: amount - amount * percent / 100  # mutare:ignore

You may add a free-text reason; it is also possible to narrow the directive to specific mutator families with a [...] filter listing the families to suppress.

# mutare:ignore nothing on the next line will be mutated

def passthrough(x), do: x + 0



def parity(n), do: rem(n, 2) == 0  # mutare:ignore[arithmetic] only `rem` will be skipped

A filter accepts any built-in family name (the full list is Mutare.Mutators.families/0 — arithmetic, relational, integer, collection, …) or any custom mutator's name/0. Filtering fails safe: an unknown name (a typo) or an empty [] matches nothing, so the mutant runs rather than being silently hidden.

Qualify a family with :label to suppress just one kind of its mutants. Here x < 0 and x <= 0 are equivalent — the boundary 0 returns 0 down either branch — so that one mutant can never be killed; suppress it while every other relational mutant (>, >=, ==, …) keeps running:

def floor_zero(x), do: if(x < 0, do: 0, else: x)  # mutare:ignore[relational:<=] 0 ≤ 0 returns 0 here

Each family's own labels are named in turn — relational → > >= < <= == != === !==, integer → zero succ pred, return_value → empty sentinel — and mix mutare --list-mutators prints every built-in family's labels. A qualified label that is not declared by a known family is a hard error.

For spans that aren't worth annotating line by line — e.g. a literal lookup table, a generated module — you can suppress a region with # mutare:ignore-start … # mutare:ignore-end (filter/reason can be provided on the start comment), or a whole file with # mutare:ignore-file:

# mutare:ignore-start spot-checked; the round-trip property test covers the whole table

def encode(?A), do: ?B

def encode(?B), do: ?C

def encode(?C), do: ?D

# mutare:ignore-end

The full grammar is in Mutare.Ignore.

Usage

mix mutare                             # mutate everything under lib/

mix mutare --only lib/billing          # scope to one path

mix mutare --since master              # only lines this branch changed

mix mutare --mutators relational       # run one built-in family

mix mutare --skip-lifting MyApp.Mod.fun/2

                                       # keep one function in-place; no guard,

                                       #   head-pattern, or clause-drop mutants

mix mutare --min-score 70              # fail below a mutation score

mix mutare --max-no-coverage 0         # fail on uncovered mutants

mix mutare --fail-on-poisoned          # fail on compile-poisoned mutants

mix mutare --fail-on-harness-error     # fail on infrastructure verdict gaps

mix mutare --per-file                  # run whole covering files, not per-test-case

mix mutare --full                      # run the whole suite per mutant

mix mutare --workers 4                 # run N mutants concurrently

mix mutare --schedulers 2              # scheduler threads per worker (default:

                                       #   the machine's ÷ workers; `all` = untrimmed)

mix mutare --timeout 30000             # per-mutant wall-clock cap, in ms

mix mutare --max-heap-mb 4096          # per-process heap cap: a mutant that

                                       #   allocates without bound dies as a

                                       #   test failure, not a host OOM

mix mutare --report json:mutare.json   # write a machine-readable report

Most projects can start without configuration. Add .mutare.exs when you want to scope the run, tune CI gates, configure routing for a project macro, or add an application-specific mutator:

[

  paths: ["lib"],

  exclude: ["lib/generated/**"],



  # Extend the built-in mutators with one of your own.

  mutators: [:builtins, MyApp.Mutators.AccessPolicy],



  # Leave DSL-only macro arguments as written, or skip a call outright.

  call_routes: [

    {Ecto.Query, :from, :raw},

    {MyApp.Schema, :field, 2, [:expression, :raw]},

    {Mixpanel, :track, 3, :skip}

  ],



  # Extend the built-in timeout table to your own functions.

  argument_marks: [{MyApp.Http, :get, 2, [{:keyword, :recv_timeout}], :timeout}],



  # Keep a compatibility-sensitive function in-place.

  skip_lifting: [{MyApp.Legacy, :parse, 1}],



  # CI gates.

  min_score: 70,

  max_no_coverage: 0,

  fail_on_poisoned: true,

  fail_on_harness_error: true,



  workers: 4,

  timeout_multiplier: 3.0,



  # Harden against flaky tests.

  baseline_runs: 2,

  baseline_retries: 1,

  kill_runs: 2,

  test_selection: :tests,



  # Emit several reports at once.

  reporters: [:human, {:json, "mutare.json"}, {:sarif, "mutare.sarif"}]

]

These are the common keys. mix help mutare documents the full option set, including sandbox reuse, run caps, retry guards, strict ignore handling, and the matching CLI flags.

Live progress

While a run is in flight, Mutare writes progress to stderr: the current phase, each survivor as soon as it appears, timeouts, harness errors, and — in a terminal — a live status block with the active mutant and an ETA. When stderr is not a terminal (a CI log, a redirect), the status block becomes a PROGRESS line with the counts and ETA, written at every tenth of the mutants and at least every two minutes, compile included; grep PROGRESS progress.log | tail -n 1 shows where a run stands. Mutare still prints the final report to stdout, so mix mutare > report.txt captures the report while progress stays visible in the terminal.

Machine-readable output

By default Mutare prints the human report to stdout. --report FORMAT[:PATH] selects another format, and the flag is repeatable:

mix mutare --report json:mutare.json --report sarif:mutare.sarif

A JSON or HTML report written to a file is kept up to date while the run is in progress, with the mutants not yet tested marked Pending, so a run killed partway still leaves its results behind. On SIGTERM, Mutare writes every report except SARIF from the results so far and exits with status 143.

When every machine format is written to a file, Mutare still prints the human report to the console; when any report is directed to stdout (no :PATH), the human report is suppressed to avoid a collision. Only one report may be directed to stdout, and no two may share a path — a second document on the same destination would corrupt or overwrite the first, so --report json --report sarif is rejected rather than run.

Routing calls: skipping calls and arguments

There are two common reasons to exclude a call from mutation. Some are not worth testing — an analytics emitter, a logger, a metrics call — and every mutant inside them is noise. Some macros take arguments that are not ordinary runtime code — a query DSL body, a pattern, a schema definition — and mutating inside those is noise too, and sometimes breaks the single metamutant compile.

Both are call_routes: entries. An entry names a call by module, function, and arity (macros and functions alike; Mutare matches remote calls, including aliases, and imports it can resolve, with or without a pipe) and specifies how to treat it:

call_routes: [

  # skip the whole call: nothing inside it is mutated, and the call itself is never rewritten

  {Mixpanel, :track, 3, :skip},



  # every argument of `from/_` is left as written

  {Ecto.Query, :from, :raw},



  # only the 2nd argument of `field/2` is left as written; the 1st mutates as normal

  {MyApp.Schema, :field, 2, [:expression, :raw]},



  # the assigns map itself is never collapsed to `%{}` (a crash, not a signal), but the values

  # inside it still mutate

  {Phoenix.Controller, :render, 3, [:expression, :expression, :interior]},



  # one option of a literal keyword argument: `recv_timeout:` is left alone, `pool:` still mutates

  {MyApp.Http, :get, 2, [:expression, [recv_timeout: :raw]]}

]

An entry is {Module, :name, arity, treatment}, or {Module, :name, treatment} to match any arity. The treatment is either :skip for the whole call, one word for every argument, or a per-position list:

A per-position list is padded with :expression, so [:expression, :raw] means "mutate the first argument, leave the second as written, mutate the rest". :skip is only ever the whole treatment; inside a list, write :raw.

Two wildcards cover the awkward cases, and more specific entries win:

call_routes: [

  # whole module

  {Sentry, :*, :skip},



  # one macro name, wherever it comes from

  {:*, :sigil_X, :raw},



  # more specific entries win: everything in MyApp.Sql is skipped except select/2

  {MyApp.Sql, :*, :skip},

  {MyApp.Sql, :select, 2, [:expression, :raw]}

]

Local-call limitation: module-specific routes do not match unqualified calls to functions defined in the calling module. For example, {SomeModule, :foobar, 0, :skip} does not skip foobar() inside SomeModule. It does skip SomeModule.foobar(), including inside SomeModule, and foobar() in another module that imports SomeModule when Mutare can resolve the import. Mutare does not resolve a local call to its defining module.

The module wildcard bypasses that limitation: {:*, :foobar, 0, :skip} matches by name and arity, so it also skips local foobar() calls inside modules that define foobar/0. It applies across modules, subject to the precedence of more specific routes described above.

That is the whole vocabulary .mutare.exs accepts. Richer DSL support belongs in a library adapter: use an extension for shape-aware routing, and host mutators for mutations inside DSL fragments. The Extending Mutare guide covers that path.

A route that matches no call anywhere in a full scan is reported as a warning, so a typo'd module or a wrong arity never sits silently inert.

Argument marks: extending the timeout table

Routes are blunt on purpose: :raw holds a position back from every mutator whatever its value. The built-in timeout handling is finer than that. A duration position (Process.sleep/1, GenServer.call/3's third argument, Task.async_stream's timeout: option, …) is marked :timeout, and each mutator handles the mark according to its mutation rules: the integer family skips integers there, the atom family skips only :infinity, and other families still apply — so a computed duration like base * 2 still mutates its 2.

argument_marks: extends those tables to your own functions, in the exact shape the mutators declare them:

argument_marks: [

  {MyApp.Cache, :put, 3, [2], :timeout},                       # a positional argument

  {MyApp.Http, :get, 2, [{:keyword, :recv_timeout}], :timeout} # a trailing-option value

]

An entry is {Module, :function, arity, positions, label}; positions lists argument indices (a value piped into the call is index 0) and {:keyword, key} option keys. The label must be one an enabled mutator declares — :timeout is built in, and a companion package documents its own — so a typo fails at startup. Reach for a route when a position should simply not mutate; reach for a mark when the reaction should depend on the value.

Choosing which mutators run

By default, every built-in mutator family runs. The :mutators option is only needed when you want to narrow that set, configure one family, or add a custom mutator.

# A mutator plus config:

mutators: [{Mutare.Mutators.Arithmetic, []}, {MyApp.Mutators.AccessPolicy, []}]



# Empty config can be omitted. Built-in family atoms expand to their modules.

mutators: [:arithmetic, MyApp.Mutators.AccessPolicy]

Use :builtins when you want to start from the default set:

mutators: [:builtins, MyApp.Mutators.AccessPolicy]   # EXTEND  — all built-ins + your own

mutators: [MyApp.Mutators.AccessPolicy]              # REPLACE — only your own



mutators: [{:builtins, except: [:arithmetic, :relational]}]   # all built-ins except these

To reconfigure a built-in, exclude the stock version and add the configured module explicitly:

mutators: [

  {:builtins, except: [:convention]},

  {Mutare.Mutators.ConventionAtom, pairs: [[:active, :inactive]]}

]

On the CLI, --mutators takes a CSV of built-in family atoms (--mutators builtins,relational). except: and custom modules are .mutare.exs-only.

Custom mutators

A custom mutator is useful when your application has a meaningful alternative outside the built-in mutation set. Suppose editing requires stricter permission than viewing: replacing Permissions.can_edit?/2 with Permissions.can_view?/2 checks whether the tests prevent a view-only user from editing.

A mutator implements Mutare.Mutator: name/0 supplies the report name, and mutate/1 returns either :skip or a list of replacement AST nodes. resolved_call/1 recognizes the call even when MyApp.Permissions is aliased, and its rebuild function preserves the form used by the source:

defmodule MyApp.Mutators.AccessPolicy do

  @behaviour Mutare.Mutator



  alias Mutare.Calls



  @impl true

  def name, do: :access_policy



  @impl true

  def mutate(node) do

    case Calls.resolved_call(node) do

      {[:MyApp, :Permissions], :can_edit?, [actor, record], rebuild} ->

        [rebuild.(:can_view?, [actor, record])]



      _ ->

        :skip

    end

  end

end

This assumes both permission functions exist with the same arity, keeping the generated mutant compile-safe. Enable it with mutators: [:builtins, MyApp.Mutators.AccessPolicy]. While you develop it, mix mutare --check --verify-invariants builds the metamutant without running tests and fails if your mutator produces a mutant the report would misrepresent, such as one the metamutant cannot run.

Example output:

mutare: 3 mutants across 1 file(s)

compiling metamutant once, baseline first…



.S.



lib/calc.ex:3  [relational, in-place]  SURVIVED

-  def gte?(a, b), do: a >= b

+  def gte?(a, b), do: a > b



mutation score: 66.7%  (2 killed, 1 survived, 3 total)

That survivor indicates that nothing in the suite distinguishes > from >= at the boundary — a missing boundary test.

Development

mix test                                      # full suite (~3 min; subprocess + property soaks)

mix test --exclude runner --exclude property  # fast loop (~4s)

mix format

mix compile --warnings-as-errors              # the project is kept warnings-clean

mix docs                                      # generate the HexDocs locally

License

MIT © Benjamin Fox