Mutare

Mutare is a mutation testing system for Elixir, that mutates the source you actually write, and compiles once.

Mutation testing measures whether your test suite actually constrains the behavior of your code: it deliberately breaks your source code one small change at a time, and each time your tests nevertheless still pass it has located a gap in your test suite.

Other Elixir mutation testing libraries, such as Muex, compile your code once per mutant, which can be quite slow, even with incremental recompilation. Mutation testing is never fast, but mutare aims to be the fastest Elixir mutation library with its compile-once approach. Mutare compiles a metamutant, a version of your program embedding all possible mutants behind runtime switches, and then selects the active mutant per test run via an environment variable.

Installation

The easiest way to install mutare is the igniter installer, which adds mutare to your :dev/:test deps and auto-configures framework integrations:

mix igniter.install mutare

It inspects your dependencies and, for each framework it finds, adds the matching companion package and wires it into a generated .mutare.exs:

Detected dependencyPackage addedWired into
:plug / :bandit / :plug_cowboy / :phoenixmutare_plug:mutators — Mutare.Plug.all/0
:phoenixmutare_phoenix:mutators — Mutare.Phoenix.all/0; :extensions — Mutare.Phoenix
:phoenix_live_viewmutare_phoenix_live_view:mutators — Mutare.Phoenix.LiveView.all/0
:ecto_sql / :phoenix_ecto / :ectomutare_ecto:mutators — {Mutare.Ecto, repo: YourRepo}
:oban / :oban_promutare_oban:mutators — Mutare.Oban.all/0
:decimalmutare_decimal:mutators — Mutare.Decimal.all/0
:swoosh / :phoenix_swooshmutare_swoosh:mutators — Mutare.Swoosh.all/0
:phoenix_swooshmutare_phoenix_swoosh:mutators — Mutare.Phoenix.Swoosh.all/0
:gettextmutare_gettext:extensions — Mutare.Gettext

A mutator package extends the :mutators list; a non-mutating extension like mutare_gettext (which teaches mutare a library's compile-time vocabulary so the built-in mutators can deal with it) joins the :extensions list; mutare_phoenix does both, since its front module also routes Phoenix's compile-time macros. The Ecto repo is detected automatically (pass --repo MyApp.Repo to override). If you already have a .mutare.exs, it is left untouched and the recommended keys are printed for you to merge in.

You can install igniter globally, with mix archive.install hex igniter_new, or add it to your project's mix.exs:

{:igniter, "~> 0.8", only: [:dev]},

Or add mutare by hand — though if you're using any macro-heavy libraries, like ecto or gettext, you'll need to add the relevant mutator or extension too.

# mix.exs

{:mutare, "~> 0.1", only: [:dev, :test], runtime: false}

# Optionally, also:

# {:mutare_ecto, "~> 0.1", only: [:dev, :test], runtime: false}

# {:mutare_decimal, "~> 0.1", only: [:dev, :test], runtime: false}

# {:mutare_gettext, "~> 0.1", only: [:dev, :test], runtime: false}

# {:mutare_plug, "~> 0.1", only: [:dev, :test], runtime: false}

# {:mutare_phoenix, "~> 0.1", only: [:dev, :test], runtime: false}

# {:mutare_phoenix_live_view, "~> 0.1", only: [:dev, :test], runtime: false}

# {:mutare_swoosh, "~> 0.1", only: [:dev, :test], runtime: false}

# {:mutare_phoenix_swoosh, "~> 0.1", only: [:dev, :test], runtime: false}

Then run mix mutare.

How it works

  1. Transform. Every in-scope source file is rewritten into a metamutant that embeds all of its mutants. Mutare transforms the code you write, before macro expansion—so any macros that don't accept arbitrary expressions will probably need their arguments routed :raw in your config (see "Routing calls" below).
  2. Compile once. The metamutant compiles a single time. The source code doesn't change between runs, so there is no per-mutant recompilation.
  3. Run the suite per mutant. A baseline test run must pass; a coverage probe then maps each mutant to the test files that exercise it. Each mutant runs in a fresh mix test OS process with MUTARE_ACTIVE_MUTANT set, :workers at a time, each capped by a wall-clock timeout.
  4. Report. Surviving mutants are listed in an abbreviated format as the run progresses, and you get a full report, with diffs and a mutation score, at the end. You can also enable JSON, HTML, or SARIF format output.

One observable difference is worth knowing: a suite that is green under plain mix test can fail during Mutare's baseline run when a test asserts exact FunctionClauseError fields or stacktrace frames, because Mutare's function lifting renames those internals. See "Troubleshooting baseline-only failures" in the mix mutare task docs (mix help mutare) for the mechanics and the skip_lifting escape hatch.

Features

Suppress a known-equivalent mutant with a comment — a trailing comment marks its line as ignored, and a standalone comment applies to the next line. Ignored mutants are excluded from the score.

def discounted(amount, percent), do: amount - amount * percent / 100  # mutare:ignore

You may add a free-text reason; it is also possible to narrow the directive to specific mutator families with a [...] filter listing the families to suppress.

# mutare:ignore nothing on the next line will be mutated

def passthrough(x), do: x + 0



def parity(n), do: rem(n, 2) == 0  # mutare:ignore[arithmetic] only `rem` will be skipped

A filter accepts any built-in family name (the full list is Mutare.Mutators.families/0 — arithmetic, relational, integer, collection, …) or any custom mutator's name/0. Filtering fails safe: an unknown name (a typo) or an empty [] matches nothing, so the mutant runs rather than being silently hidden.

Qualify a family with :label to suppress just one kind of its mutants. Here x < 0 and x <= 0 are equivalent — the boundary 0 returns 0 down either branch — so that one mutant can never be killed; suppress it while every other relational mutant (>, >=, ==, …) keeps running:

def floor_zero(x), do: if(x < 0, do: 0, else: x)  # mutare:ignore[relational:<=] 0 ≤ 0 returns 0 here

Each family names its own labels — relational → > >= < <= == != === !==, integer → zero succ pred, return_value → empty sentinel — and mix mutare --list-mutators prints every built-in family's labels. A qualified label that a known family doesn't declare is a hard error with a "did you mean", so a typo can't slip through as a silent no-op.

For spans that aren't worth annotating line by line — a literal lookup table, a generated module — suppress a region with # mutare:ignore-start … # mutare:ignore-end (both take the same filter and reason, carried on the -start), or a whole file with # mutare:ignore-file:

# mutare:ignore-start spot-checked; the round-trip property test covers the whole table

def encode(?A), do: ?B

def encode(?B), do: ?C

def encode(?C), do: ?D

# mutare:ignore-end

The full grammar is in Mutare.Ignore.

Usage

mix mutare                             # mutate everything under lib/

mix mutare --only lib/billing          # scope to one path

mix mutare --since master              # only lines changed vs a git ref

mix mutare --mutators relational       # run one built-in family

mix mutare --skip-lifting MyApp.Mod.fun/2

                                       # keep one function in-place; no guard,

                                       #   head-pattern, or clause-drop mutants

mix mutare --min-score 70              # fail below a mutation score

mix mutare --max-no-coverage 0         # fail on uncovered mutants

mix mutare --fail-on-poisoned          # fail on compile-poisoned mutants

mix mutare --fail-on-harness-error     # fail on infrastructure verdict gaps

mix mutare --per-file                  # run whole covering files, not per-test-case

mix mutare --full                      # run the whole suite per mutant

mix mutare --workers 4                 # run N mutants concurrently

mix mutare --timeout 30000             # per-mutant wall-clock cap, in ms

mix mutare --max-heap-mb 4096          # per-process heap cap: a mutant that

                                       #   allocates without bound dies as a

                                       #   test failure, not a host OOM

mix mutare --report json:mutare.json   # write a machine-readable report

Most projects can start without configuration. Add .mutare.exs when you want to scope the run, tune CI gates, teach Mutare about a project macro, or add an application-specific mutator:

[

  paths: ["lib"],

  exclude: ["lib/generated/**"],



  # Extend the built-in mutators with one of your own.

  mutators: [:builtins, MyApp.Mutators.AccessPolicy],



  # Leave DSL-only macro arguments as written, or skip a call outright.

  call_routes: [

    {Ecto.Query, :from, :raw},

    {MyApp.Schema, :field, 2, [:expression, :raw]},

    {Mixpanel, :track, 3, :skip}

  ],



  # Extend the built-in timeout table to your own functions.

  argument_marks: [{MyApp.Http, :get, 2, [{:keyword, :recv_timeout}], :timeout}],



  # Keep a compatibility-sensitive function in-place.

  skip_lifting: [{MyApp.Legacy, :parse, 1}],



  # CI gates.

  min_score: 70,

  max_no_coverage: 0,

  fail_on_poisoned: true,

  fail_on_harness_error: true,



  workers: 4,

  timeout_multiplier: 3.0,



  # Harden against flaky tests.

  baseline_runs: 2,

  baseline_retries: 1,

  kill_runs: 2,

  test_selection: :tests,



  # Emit several reports at once.

  reporters: [:human, {:json, "mutare.json"}, {:sarif, "mutare.sarif"}]

]

These are the common keys. mix help mutare documents the full option set, including sandbox reuse, run caps, retry guards, strict ignore handling, and the matching CLI flags.

Live progress

While a run is in flight, Mutare writes progress to stderr: the current phase, each survivor as soon as it appears, timeouts, harness errors, and — in a terminal — a live status block with the active mutant and an ETA. The final report still prints to stdout, so mix mutare > report.txt captures the report while progress stays visible in the terminal.

Machine-readable output

By default Mutare prints the human report to stdout. --report FORMAT[:PATH] selects another format, and the flag is repeatable:

mix mutare --report json:mutare.json --report sarif:mutare.sarif

When every machine format is written to a file, the human report still prints to the console; when any report takes stdout (no :PATH), the human report is suppressed to avoid a collision. Only one report may take stdout, and no two may share a path — a second document on the same destination would corrupt or overwrite the first, so --report json --report sarif is rejected rather than run.

Routing calls: skipping calls and arguments

Two kinds of call want leaving alone. Some are not worth testing — an analytics emitter, a logger, a metrics call — and every mutant inside them is noise. Some macros take arguments that are not ordinary runtime code — a query DSL body, a pattern, a schema definition — and mutating inside those is noise too, and sometimes breaks the single metamutant compile.

Both are call_routes: entries. An entry names a call by module, function, and arity (macros and functions alike; Mutare matches it however it is written — directly, aliased, imported, or piped) and says how to treat it:

call_routes: [

  # skip the whole call: nothing inside it is mutated, and the call itself is never rewritten

  {Mixpanel, :track, 3, :skip},



  # every argument of `from/_` is left as written

  {Ecto.Query, :from, :raw},



  # only the 2nd argument of `field/2` is left as written; the 1st mutates as normal

  {MyApp.Schema, :field, 2, [:expression, :raw]},



  # the assigns map itself is never collapsed to `%{}` (a crash, not a signal), but the values

  # inside it still mutate

  {Phoenix.Controller, :render, 3, [:expression, :expression, :interior]},



  # one option of a literal keyword argument: `recv_timeout:` is left alone, `pool:` still mutates

  {MyApp.Http, :get, 2, [:expression, [recv_timeout: :raw]]}

]

An entry is {Module, :name, arity, treatment}, or {Module, :name, treatment} to match any arity. The treatment is either :skip for the whole call, one word for every argument, or a per-position list:

A per-position list is padded with :expression, so [:expression, :raw] means "mutate the first argument, leave the second as written, mutate the rest". :skip is only ever the whole treatment; inside a list, write :raw.

Two wildcards cover the awkward cases, and more specific entries win:

call_routes: [

  # whole module

  {Sentry, :*, :skip},



  # one macro name, wherever it comes from

  {:*, :sigil_X, :raw},



  # more specific entries win: everything in MyApp.Sql is skipped except select/2

  {MyApp.Sql, :*, :skip},

  {MyApp.Sql, :select, 2, [:expression, :raw]}

]

That is the whole vocabulary .mutare.exs accepts. Richer DSL support belongs in a library adapter: use an extension for shape-aware routing, and host mutators for mutations inside DSL fragments. The Extending Mutare guide covers that path.

A route that matches no call anywhere in a full scan is reported as a warning, so a typo'd module or a wrong arity never sits silently inert.

Argument marks: extending the timeout table

Routes are blunt on purpose: :raw holds a position back from every mutator whatever its value. The built-in timeout handling is finer than that. A duration position (Process.sleep/1, GenServer.call/3's third argument, Task.async_stream's timeout: option, …) is marked:timeout, and each mutator decides what the mark means: the integer family declines any integer there, the atom family declines only :infinity, and every other family proceeds — so a computed duration like base * 2 still mutates its 2.

argument_marks: extends those tables to your own functions, in the exact shape the mutators declare them:

argument_marks: [

  {MyApp.Cache, :put, 3, [2], :timeout},                       # a positional argument

  {MyApp.Http, :get, 2, [{:keyword, :recv_timeout}], :timeout} # a trailing-option value

]

An entry is {Module, :function, arity, positions, label}; positions lists effective argument indices (a piped receiver is index 0) and {:keyword, key} option keys. The label must be one some enabled mutator understands — :timeout is built in, and a companion package documents its own — so a typo fails at startup. Reach for a route when a position should simply not mutate; reach for a mark when the reaction should depend on the value.

Choosing which mutators run

By default, every built-in mutator family runs. The :mutators option is only needed when you want to narrow that set, configure one family, or add a custom mutator.

# A mutator plus config:

mutators: [{Mutare.Mutators.Arithmetic, []}, {MyApp.Mutators.AccessPolicy, []}]



# Empty config can be omitted. Built-in family atoms expand to their modules.

mutators: [:arithmetic, MyApp.Mutators.AccessPolicy]

Use :builtins when you want to start from the default set:

mutators: [:builtins, MyApp.Mutators.AccessPolicy]   # EXTEND  — all built-ins + your own

mutators: [MyApp.Mutators.AccessPolicy]              # REPLACE — only your own



mutators: [{:builtins, except: [:arithmetic, :relational]}]   # all built-ins except these

To reconfigure a built-in, exclude the stock version and add the configured module explicitly:

mutators: [

  {:builtins, except: [:convention]},

  {Mutare.Mutators.ConventionAtom, pairs: [[:active, :inactive]]}

]

On the CLI, --mutators takes a CSV of built-in family atoms (--mutators builtins,relational). except: and custom modules are .mutare.exs-only.

Custom mutators

A custom mutator is useful when your application has a meaningful alternative that a general-purpose tool cannot know. Suppose editing requires stricter permission than viewing: replacing Permissions.can_edit?/2 with Permissions.can_view?/2 checks whether the tests prevent a view-only user from editing.

A mutator implements Mutare.Mutator: name/0 supplies the report name, and mutate/1 returns either :skip or a list of replacement AST nodes. resolved_call/1 recognizes the call even when MyApp.Permissions is aliased, and its rebuild function preserves the form used by the source:

defmodule MyApp.Mutators.AccessPolicy do

  @behaviour Mutare.Mutator



  alias Mutare.Calls



  @impl true

  def name, do: :access_policy



  @impl true

  def mutate(node) do

    case Calls.resolved_call(node) do

      {[:MyApp, :Permissions], :can_edit?, [actor, record], rebuild} ->

        [rebuild.(:can_view?, [actor, record])]



      _ ->

        :skip

    end

  end

end

This assumes both permission functions exist with the same arity, keeping the generated mutant compile-safe. Enable it with mutators: [:builtins, MyApp.Mutators.AccessPolicy].

Example output:

mutare: 3 mutants across 1 file(s)

compiling metamutant once, baseline first…



.S.



lib/calc.ex:3  [relational, in-place]  SURVIVED

-  def gte?(a, b), do: a >= b

+  def gte?(a, b), do: a > b



mutation score: 66.7%  (2 killed, 1 survived, 3 total)

That survivor says: nothing in the suite distinguishes > from >= at the boundary — a missing boundary test.

Development

mix test                                      # full suite (~3 min; subprocess + property soaks)

mix test --exclude runner --exclude property  # fast loop (~4s)

mix format

mix compile --warnings-as-errors              # the project is kept warnings-clean

mix docs                                      # generate the HexDocs locally

License

MIT © Benjamin Fox