mutagen_ex

Mutation testing for Elixir. mix mutagen mutates a chosen scope, runs a cited test set against each mutation, and emits a single JSON document classifying every mutation as killed, survived, timeout, error, or compile_error.

The CLI (mix mutagen) is the supported user-facing entry point; MutagenEx.MutationRunner.run/1 is the equivalent library-caller entry point. The CLI ends each run with an aggregate JSON document. --stream writes per-site events to the same destination before it.

Audit mode

Use audit mode to find tests that execute code without asserting its behavior. Run a whole module and read the per-function table. It lists up to 50 evaluated functions, lowest kill rate first, with zero first.

MIX_ENV=test mix mutagen --scope MyApp.Billing --tests auto --json billing.json --summary billing.md
jq '.summary.by_function[]' billing.json

Run each source file in a directory and rank the reported modules by kill rate, with zero first and unevaluated runs last.

find lib/my_app -name '*.ex' | while IFS= read -r file; do
  MIX_ENV=test mix mutagen --scope "$file" --tests auto --json "audit/$file.json" --summary "audit/$file.md"
done
find audit -name '*.json' -exec cat {} + | jq -rs 'sort_by(.summary.counts.kill_rate // 2)[] | [.summary.counts.kill_rate, ([.scope[].module] | join(", "))] | @tsv'

How much to trust a survivor

A blind measurement at a pre-release commit of this series covered ten files from two private Phoenix applications. A seeded script selected the files. They included two contexts, two controllers, LiveViews, and plain modules. A reviewer judged each survivor after seeing only the code change and the test file.

The reviewer judged public functions by the inputs their specs admit. For private functions, controllers, and LiveViews, the reviewer used the inputs their real callers, routes, and forms can deliver.

Survivors Real test gaps Share 95% interval
High confidence 27 of 32 84% 68% to 93%
High confidence, reachable by current production callers 23 of 32 72% 55% to 84%
Low confidence 8 of 24 33% 18% to 53%
Confidence Mutator Real test gaps
High boolean 7 of 7
High case_drop 12 of 15
High else_removal 3 of 3
High result_tuple 2 of 2
High compare 1 of 1
High literal 2 of 4
Low pipeline 5 of 19
Low guard_drop 2 of 4
Low with_swap 1 of 1

Every figure uses one test file per module and measures gaps in the selected tests. A follow-up ran nine of the real-gap survivors against every test file that mentions the function. Other test files in the same suite killed six of them. Three survived.

A survivor means “the tests you selected miss it”. First rerun with a wider --tests selection before writing a test.

The sample is small and per-mutator confidence is provisional. Mutagen gives advice for the current run. See the advisory decision.

This measurement predates two changes that make a whole-file run mutate more code: sites on function-head lines (guards) now run, and sites that line coverage misses are confirmed by an execution probe and then run. On the two files rerun after those changes, every earlier site kept its status and the run gained sites: 30 became 42 on one file and 42 became 69 on the other, with 30 of the 50 new sites surviving. Those added survivors have not been judged, so the shares above describe the earlier set of sites only.

Changed-code measurement before the fixes

A seeded script chose 12 real commits from two private Phoenix applications. A reviewer judged them blind, seeing only the change and the tests that ran.

What the summary showed Entries Real gaps Covered by a test that did not run False claim Noise
Check first (not executed) 3 0 1 2 0
Worth a look (high confidence) 3 0 0 0 3
JSON only (low confidence) 8 1 1 0 6

This measurement predates probe confirmation, the changed-line rule, and opt-in escalation. Those changes address the false execution claims, the off-diff survivors, and the noise from widening the test selection by default.

Third-pass measurement after the fixes

The third pass used the same 12 commits and the same blind protocol after probe confirmation, the changed-line rule, and opt-in escalation.

The report still included pipeline in Worth a look during this measurement. Five of the eight low-confidence entries came from pipeline. None was a real gap.

On a commit that came with tests, a changed-code run is usually silent. A listed survivor is more often an unobservable or unreachable change than a missing test. One of the 12 survivors here was a real test gap. Read the line before drawing a conclusion. The whole-file audit figures above support looking for test gaps.

Changed-code runs

Changed-code runs are quiet by design. On a commit that came with tests, “nothing to check” is the normal and correct output. The summary shows functions whose sites the selected tests did not execute, confirmed by probes, functions where at least three high-confidence mutants ran and all evaluated mutants survived, and survivors whose own source spans overlap changed lines. The detailed JSON holds the remaining results.

MIX_ENV=test mix mutagen --since origin/main --json out/mutagen.json --summary out/mutagen.md

Mutagen compares the working tree with the merge base of the ref and HEAD. A shallow checkout always aborts. Fetch full Git history and use fetch-depth: 0 in GitHub Actions. The run plans mutations in named runtime functions on touched lines, then compares their group fingerprints with the base source. Formatting, comments, and moved unchanged functions contribute no runtime sites. Use --scope to narrow the diff further. An empty runtime plan finishes before host setup.

The run includes untracked, unignored files. It excludes test roots and test support from source planning. An incomplete run returns exit 0 with its reasons. A tool abort returns exit 2.

Automatic test selection looks for a/b_test.exs under the configured test roots for a/b.ex under a configured source root that has a planned runtime mutation. When a --since run has runtime sites, it also selects new test files and changed test files whose parsed code differs from the merge base. Formatting and comment edits alone do not add a test file to this selection. An empty runtime plan selects no tests. Add --escalate to search for more tests for surviving or uncovered sites. Mutagen searches uncited test files for the full module name, or for the module's last segment as a whole word in a file that aliases its parent. Brace aliases and aliases declared with as: count in the module pass. It then searches for qualified function calls or captures, or bare calls where the matching module is imported or aliased. It counts text mentions and uses path order to break ties. Each step takes up to 20 files per function by default. Mutagen tests each candidate on the original code. It allows ten times the narrow baseline duration, with a thirty-second floor capped by the host boot and mutant timeouts. Each step reports candidate_deadline_ms. If a batch fails, Mutagen keeps the files that pass together in score order. The module and function passes share a two-minute budget, including candidate probes. Mutagen estimates each pass from the candidate baselines and reports files that exceed the budget as escalation_budget_exhausted. Affected survivors carry selection.escalation_incomplete: true. When escalation takes more than half the run, summary notes give the time spent in each pass. A pass judges mutants with the files admitted in that pass. Only survivors and uncovered sites advance. A timeout stays unevaluated and keeps the step that recorded it.

meta.since records the ref, merge base, and site counts. meta.complete and meta.incomplete_reasons record missing evidence from skipped or unreadable paths and sites that the runner dropped. Every abort sets meta.complete to false and adds its reason to meta.incomplete_reasons. A partial run stopped by --budget-ms records mutation_budget_exhausted. An accounting mismatch between planned sites and the final outcome records also marks the report incomplete. meta.since.unmapped lists changed code without runtime sites and omits blank, comment and structural lines. mutation.pending shows sites with no outcome, including uncovered sites and covered sites left unrun. Coverage includes green tests added during escalation. meta.tests_selected lists why each test file joined, and each result lists the steps it ran. Warnings name rejected files and give a reason. Selected and rejected test paths follow --no-source. Each step counts files dropped after a load attempt in dropped_after_load. Result selection includes after_dropped_file: true for verdicts made in that step or later. Those verdicts can reflect side effects the tool could not undo, such as writes to persistent terms. With --stream, changed-code and automatic runs emit final results and compile errors after all passes with the same fields as the aggregate. Pending sites appear in the final JSON document.

Site IDs are stable within an unchanged function. Editing a function can reassign IDs, so use them to inspect the current report.

Why mutagen_ex (and how it differs from muzak)

Elixir already has a mature mutation tester: muzak / muzak_pro by Devon Estes. If you want a polished, human-facing mutation-testing workflow for everyday TDD, muzak is the established choice and mutagen_ex is not trying to replace it.

mutagen_ex is built for repeatable audits and automated review. It gives people a Markdown summary and gives scripts and agents a stable JSON report with site IDs they can compare across runs.

mutagen_ex focuses on these workflows:

The mutator catalog and its lineage

mutagen_ex ships ten AST mutators (.spec/specs/mutators.spec.md). They fall into two groups: four that are direct Elixir realizations of the classic, language-agnostic mutation operators studied since the Mothra era, and six that are Elixir-idiomatic and have no clean parallel in the C/Java mutation literature.

The first group maps onto the selective mutation operator set — the small family Offutt et al. found sufficient to retain almost all of a full operator set's fault-detection power at a fraction of the mutant count (Offutt, Lee, Rothermel, Untch & Zapf, "An Experimental Determination of Sufficient Mutant Operators," ACM TOSEM 5(2), 1996, doi:10.1145/227607.227610; see also Jia & Harman's survey, IEEE TSE 37(5), 2011, doi:10.1109/TSE.2010.62 for the broader taxonomy these abbreviations come from):

mutagen_ex mutator Standard operator What it does
arith AOR — Arithmetic Operator Replacement swaps + ↔ -, * ↔ / on numeric binary ops
compare ROR — Relational Operator Replacement swaps == ↔ !=, < ↔ >=, > ↔ <=
boolean LCR — Logical Connector Replacement swaps and ↔ or, && ↔ ||, drops not/!
literal CRP — Constant Replacement flips true ↔ false, swaps small integer literals

The remaining six mutate constructs that are idiomatic to Elixir/Erlang and have no language-agnostic equivalent in the standard catalogs — they exist because the standard operators say nothing about tagged tuples, multi-clause guards, with, or the pipe:

mutagen_ex mutator What it does Why it has no classic parallel
result_tuple flips {:ok, x} ↔ {:error, x} targets Elixir's tagged-tuple result convention
guard_drop drops a function-clause when guard guards are an Elixir/Erlang dispatch construct
with_swap swaps the first two leading <- clauses of a with with is an Elixir control structure
else_removal drops the else branch of if/with shaped to Elixir's if/with else semantics
case_drop drops the last case/cond clause closest to the classic statement/clause-deletion (SDL) family, but Elixir's exhaustiveness and CaseClauseError behaviour make it its own thing (see Known limitations)
pipeline reorders adjacent |> segments the pipe operator is Elixir-idiomatic

The split is deliberate: the four classic operators give the catalog a well-understood, citable backbone, while the six idiomatic operators cover the fault classes that only show up in Elixir code. Treat the standard-operator names above as the vocabulary a reviewer (human or LLM judge) already knows; treat the idiomatic six as the Elixir-specific extension that vocabulary lacks.

Mutator confidence

Every aggregate mutation.results[] entry and every NDJSON "result" line has a "confidence" field. The ratings apply to whole operators and draw on manual review of survivors from two Phoenix apps. Even a high-rated survivor can be equivalent to the original code.

The ratings are provisional. We will re-measure each mutator as more files are judged.

Mutator Confidence Reason
arith high The swapped operator can change a numeric result.
compare high Negating the comparison can change which branch runs.
boolean high Swapping a connector or removing negation can change the value or short-circuit behavior.
literal high A caller can observe the changed value or pattern.
with_swap low Two checks can still return the same failure in practice, or their failures may never occur together.
case_drop high Inputs handled by the removed final arm can now raise a clause error.
pipeline low Independent stages often give the same result in either order, as with LiveView assigns or Map.put/3 calls on different keys.
result_tuple high The tag swap changes which consumers treat a result as success or failure.
else_removal high Removing the branch changes how a false condition or failed match is handled.
guard_drop low Callers often cannot supply a value that the removed guard would have rejected.

with_swap applies its failure-priority filters only when the two clauses are independent. It treats a pair as dependent when a name bound by either pattern appears anywhere in the other clause, or when either clause uses binding(), dbg() or __ENV__. It records skip reasons in mutation.skipped.

Reason When it applies
bound_var_used_before_binding A variable-shaped name in the moved-first expression also appears as a binding in the now-later pattern. The check ignores module attributes and binary modifiers.
insufficient_refutable_patterns In an independent pair, either match uses an unguarded variable, a wildcard, or a literal pattern with an equal literal expression.
indistinguishable_failures In an independent pair, both expressions are the same literal or read the same variable.

The RHS name check can skip a valid swap when the name has an outer binding or appears only inside a nested binder or quote. This includes names local to fn, case, for, receive, rescue, nested with, match?, matches and destructure. The compiler resolves dependencies in guards, pins and binary sizes for kept swaps. Unexpanded macros can hide reads and bindings. An else arm does not affect validation. Even if it ignores the failure value, swapping the clauses can run side effects or raise an exception that the original order avoided.

Self-mutation

The repository tests mutation runs against the mutator registry, scope resolver, and JSON reporter. The harness copies those modules under a separate namespace in a temporary Mix project and runs mix mutagen from a fresh shell. Run it with mix test --include self_mutation. Self-mutation test.

Install

Mutagen requires Elixir 1.18 or later and Erlang/OTP 27 or later.

System tool

Install the Mix task for your current Elixir installation. Mix fetches the source from Hex and builds the archive with your local toolchain.

mix archive.install hex mutagen_ex
mix help mutagen

To pin a version in CI, use mix archive.install hex mutagen_ex 0.2.0 --force.

Project dependency

Add this entry to deps/0 in your project's mix.exs:

{:mutagen_ex, "~> 0.2", only: [:dev, :test], runtime: false}
mix deps.get
MIX_ENV=test mix help mutagen

With either install, run the task from the host project:

MIX_ENV=test mix mutagen --scope lib/foo.ex --tests test/foo_test.exs --json report.json

Host test setup

Run MIX_ENV=test mix mutagen with either install method. This loads config/test.exs and compiles test support. Explicit environments such as MIX_ENV=ci are allowed when the host configures them for tests. A host can also set def cli, do: [preferred_envs: [mutagen: :test]] in mix.exs. Mix may print dependency discovery output before the task runs.

After compiling, the task loads runtime config and runs the host test alias entries before the test task. A Phoenix alias can create and migrate the test database there. If the alias lacks those steps or you use --no-test-alias, create and migrate the test database before invoking mutagen. Pass MIX_TEST_PARTITION if your configuration uses it to select a database. Mutagen ignores arguments on the test entry and uses --tests and the helper's ExUnit configuration for selection. Alias functions get []. Mutagen ignores their return values. An alias with entries and no recognizable test entry produces a report warning.

The task starts the host application and requires test_helper.exs once from each configured :test_paths entry, with ExUnit autorun disabled. The default is test/test_helper.exs when test/ exists. Run from the app that owns the tests in an umbrella project. --no-host-app or MUTAGEN_NO_HOST_APP=1 skips application startup and still loads runtime config and the helper.

Coverage, baseline, and every mutant use the helper's ExUnit options. Cases retain their declared async value. With max_cases: 1, ExUnit runs cases serially in its usual async-then-sync order. Whole-file targets honor helper includes and excludes. An explicit tag or line target takes precedence when narrowing the test set. A coverage or baseline pass that executes no tests aborts with no_tests_run. Helper formatters run on every pass. A formatter that writes a file may overwrite it after each mutant. Mutagen writes the final report before running ExUnit after_suite callbacks. Each callback gets its own --boot-timeout-ms deadline. Cleanup errors and timeouts go to stderr. A callback that calls System.halt/1 decides the VM exit code.

--boot-timeout-ms bounds runtime configuration, alias pre-steps, application startup, and helper evaluation after compilation. Coverage and baseline each get the greater of --boot-timeout-ms and ten times --timeout-ms. The mutation budget starts after baseline. An OS process started by a test-alias command can outlive a boot timeout. A helper that calls System.halt/1 ends the BEAM before reporting.

Abort documents include a fixed hint and, where available, the step, exception type, file and line. Host diagnostics go to stderr. Do not publish stderr as a CI artifact. Abort documents repeat this warning in details.note.

--json PATH is the guaranteed report channel. The file contains exactly one JSON document, or NDJSON with --stream, regardless of host output.

Without --json, Mutagen writes the report to stdout and routes the :user device and known console logger handlers to stderr. The VM can write directly to file descriptor 1, including when an OTP logger handler crashes. In-process I/O routing cannot intercept those writes. Mutagen prints a reminder on stderr when it writes to stdout. Use --json PATH and --summary PATH in CI.

Mutagen writes the file with mode 0600. If it cannot safely write the file, it warns on stderr and falls back to stdout.

Entrypoints

mutagen_ex boots MutagenEx.Application whenever the :mutagen_ex application starts. That supervisor is the program entrypoint; everything the tool runs lives under MutagenEx.TaskSup.

User-facing entrypoint surfaces:

Only one concurrent MutagenEx mutation cycle per BEAM is supported; concurrent callers are refused with {:error, :cover_already_running, _}. See .spec/decisions/supervision_tree.md.

What mix mutagen does

For the selected scopes, mutagen_ex:

  1. Resolves the target to one or more (module, file) pairs.
  2. Runs the cited tests to record line-level coverage for the in-scope modules.
  3. Enumerates mutation sites from each in-scope file's AST. It uses line coverage first and probes remaining runtime sites before reporting them as unexecuted.
  4. Runs the cited tests on the original code to check the baseline. A red baseline aborts the run.
  5. For each site, applies the mutator, recompiles the module in place, re-runs the cited tests, and classifies the result.
  6. Restores the original module after every site.
  7. Emits the final JSON document.

Tests run serially within each pass. Mutation sites run serially by default in the same BEAM VM. --max-concurrency enables experimental parallel dispatch.

Mutagen applies changes in memory via Code.compile_quoted/2 and restores the original module after every site. The byte-identity contract covers lib/, _build/, cover/, host project config (mix.exs, mix.lock, .formatter.exs), and tmp entries with the mutagen_ex_ prefix — see mutagen.mutation_pipeline.r11 and mutagen.coverage.r7.

Basic usage

# Single file scope, single test file
MIX_ENV=test mix mutagen --scope lib/foo.ex --tests test/foo_test.exs

# Module scope, tag-cited tests, custom per-mutation budget
MIX_ENV=test mix mutagen --scope MyApp.Foo --tests tag:fast --timeout-ms 10000

# Function scope, deterministic seed, redirected output
MIX_ENV=test mix mutagen --scope MyApp.Foo.bar/1 \
            --tests test/foo_test.exs \
            --seed 42 \
            --json out/mutagen.json

A survivor means “the tests you selected miss it”. First rerun with a wider --tests selection before writing a test.

Reading the summary

Add --summary mutagen.md for a Markdown file you can read or paste into a pull request. The JSON always includes the same findings under summary. Both output paths use the project-root boundary by default.

The headline leads with the highest-priority finding. summary.checked reports full or partial along with the total sites, uncovered sites, and sites left unevaluated for other reasons. A partial run states how many sites it could not check. When it has no finding, the headline says “Mutagen found nothing in the sites it could check.”

Copy the printed MIX_ENV=test mix mutagen --site … command from the summary to rerun the one site with --tests auto --escalate and up to 40 test files per function in each pass. The escalation cap is the larger of 30 seconds and ten times the narrow baseline. A smaller configured budget still applies. The summary says when escalation admitted no files. Changed-code reruns keep the merge base from the report. Run the command from the same project checkout. Broader tests can kill a site that previously survived. With --no-source, the summary omits file names, function names and excerpts and prints an opaque reference for the rerun.

Running in CI

Mutagen is advisory. Each run reports the current checkout and uses no repository state from previous runs. A shallow checkout aborts because the diff needs the merge base. Use fetch-depth: 0. The unmapped list names changed lines Mutagen could not map to a runtime site, such as attributes and macros. A separate list shows changed lines outside an explicit --scope. The summary names changed lines that tests missed inside evaluated sites and reports partial coverage. Use --json PATH and --summary PATH in CI. Upload the JSON file and keep stderr out of artifacts.

name: Mutation advice
on: pull_request

jobs:
  mutagen:
    runs-on: ubuntu-latest
    permissions:
      contents: read
    steps:
      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
        with:
          fetch-depth: 0
          persist-credentials: false
      - uses: erlef/setup-beam@54075bcc5e249e4758d363f27d099f55d843f124 # v1.24.1
        with:
          version-file: .tool-versions
          version-type: strict
      - name: Collect mutation advice
        continue-on-error: true
        env:
          MUTAGEN_BASE: ${{ github.event.pull_request.base.sha }}
        run: |
          MIX_ENV="test" mix deps.get
          MIX_ENV=test mix mutagen --since "$MUTAGEN_BASE" --json mutagen.json --summary mutagen.md
      - name: Add advice to the job summary
        if: always()
        run: |
          if [ -f mutagen.md ]; then
            cat mutagen.md >> "$GITHUB_STEP_SUMMARY"
          fi
      - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
        if: always()
        with:
          name: mutagen
          path: mutagen.json

A successful tool run exits 0 even when tests miss mutations. The continue-on-error setting also allows the job to proceed after a tool abort. The example assumes the project declares the dev/test dependency. The job summary includes the abort reason and its hint. To post the Markdown on a same-repository pull request, grant pull-requests: write to the job and set GH_TOKEN and PR_NUMBER for an optional step. Run gh pr comment "$PR_NUMBER" --body-file mutagen.md when the file exists.

This repository runs its normal tests and host fixtures in CI. Mutagen refuses a changed-code run on its own runtime modules.

Set --timeout-ms for scopes whose tests finish quickly. A runaway mutant can consume the full timeout. Set --budget-ms to stop dispatching new mutants once the mutation phase has used its budget. Use meta.timings_ms to size both limits on your CI runner. Dividing mutation time by the number of attempted sites gives the average cost per mutant.

Flags

Flag Purpose
--summary <path> Write a Markdown summary to a separate file. As with --json, the path must stay within the project root unless --unsafe-json-outside-project is set.
--summary-max <int> Maximum entries in Worth a look. Default 5. Zero keeps only the omitted count.
--site <id> Rerun one site using the report ID or opaque reference.
--scope <target> What to mutate. A .ex file path, a module name, or Module.fun/arity. Required unless you use --since or --site. Repeatable up to 100 occurrences.
--tests <target> A _test.exs path, a file:line pair, or tag:<name>. Repeatable up to 100 times. Use auto on its own. Required unless you use --since or --site.
--boot-timeout-ms <int> Deadline in milliseconds for startup after compilation. Default 300000.
--no-source Omit source excerpts, excerpt lines, file and function fields from mutation results and compile errors, including stream entries.
--include-host-output Include captured compiler and test stderr and abort messages, with best-effort redaction.
--no-test-alias Skip entries in the host's test alias before the test task.
--no-host-app Load runtime configuration and the helper without starting the host application.
--since <ref> Plan changed runtime sites from the merge base. Defaults to --tests auto.
--escalate Widen automatic test selection for surviving or uncovered sites. Off by default.
--no-escalate Accepted with no effect for one release.
--escalate-max-files <int> Maximum test files per function in each extra pass. Default 20.
--escalate-budget-ms <int> Total time for the extra passes. Default 120000. Zero removes the limit for runs without --site. A single-site run caps the budget at the larger of 30 seconds and ten times the narrow baseline.
--timeout-ms <int> Override the per-mutant deadline in milliseconds. The default uses three times the baseline duration with a 5000 ms floor. Mutagen warns if a requested timeout is shorter than the baseline.
--no-fail-fast Run the full cited suite for each mutant. By default, mutation runs stop executing tests at the first failure.
--seed <int> ExUnit seed, propagated to every test-running phase. Default 0. Controls test ordering only, not mutation enumeration order.
--json <path> Write the final JSON document to <path> instead of stdout. The document always ends with a single newline. Path is canonicalised before any mutation runs: .. segments and NUL bytes are refused at parse time, and the resolved path must stay inside the project root unless --unsafe-json-outside-project is also passed.
--unsafe-json-outside-project Opt-in to writing --json output outside the project root. CI integrations targeting an artifacts directory above the project root pass this; everyday use should leave it off. Emits a one-shot stderr warning naming the resolved target at run start.
--max-sites <int> Upper bound on enumerated mutation sites for one run. Default 10000. Exceeding the cap aborts with abort_reason: "too_many_sites" BEFORE the mutation runner starts — narrow --scope (or raise the cap) to proceed.
--budget-ms <int> Optional aggregate wall-clock budget for the mutation phase, in milliseconds. Default unbounded (--timeout-ms still bounds each site). When elapsed, the runner stops dispatching new sites and emits a truncated: true partial JSON report.
--max-concurrency <int> Cap on the number of per-site mutation tasks Task.Supervisor.async_stream_nolink/4 runs in parallel. Default 1 (serial execution). Values greater than 1 are EXPERIMENTAL and emit a one-shot stderr warning because real ExUnit/:cover backends can produce incorrect kill/survive classification and corrupted coverage. See Parallel mode below.
--stream Emit one NDJSON line per completed site (and start/end envelope lines) to the same sink the aggregate document goes to. Off by default.
--no-progress Suppress the human-readable per-site progress feed on stderr. Default is auto-on when stderr is a TTY, auto-off otherwise.

See Known limitations for rejected options such as --no-json and --scope file.ex:Module.

Per-mutant fail-fast

By default, a mutant is killed when ExUnit records a test failure before the per-mutant deadline. This includes a mutant whose full suite would time out in a later test. If the deadline arrives before any failure, the mutant is classified as a timeout. Baseline and coverage have no failure cap. JSON reports the mode as meta.fail_fast and reports true if argument parsing fails before a mode is available.

Fail-fast preserves survivors and existing kills when tests run independently. If one test sets up or cleans up state for another, a failed run can skip that test and change the next mutant's verdict, even if the next mutant runs every test. These suites already depend on ExUnit's test ordering when the seed changes. Use --no-fail-fast to run the full suite per mutant. Skipped tests can also change per-mutant warnings and tainted_predecessors. A shorter mutation phase can complete more sites within the same --budget-ms.

Mutagen sets the seed, serial execution, and the failure cap after test files load. Each phase keeps the loaded host formatters and adds its own failure formatter when needed. Other host options, such as test timeouts and log capture, stay active. The resolved test filter supplies include and exclude for each phase. Mutagen restores the host settings after each phase.

Parallel mode, streaming, and progress

Mutagen includes the wiring for parallel per-site dispatch, NDJSON streaming, and a per-site progress feed. The mechanism is in place, but --max-concurrency > 1 is EXPERIMENTAL: the in-process pipeline shares ExUnit's global server, the Code.Server's per-module load locks, and :cover instrumentation across all per-site tasks (see "Known limitations" below). On real ExUnit/:cover backends, parallel dispatch may produce incorrect kill/survive classification and corrupted coverage. Every report records meta.max_concurrency. Values above 1 emit a stderr warning and add "Verdicts are unreliable in experimental parallel mode." to the report's top-level warnings. The default is 1.

Mutagen has no runtime dependencies — it installs cleanly as a dependency-free Mix archive (mix archive.install). Per-site observation (NDJSON streaming and the progress feed below) rides the mutation runner's single :on_site_completed callback rather than a :telemetry event surface.

NDJSON streaming (--stream)

When --stream is set, MutagenEx.JsonStreamer emits one JSON object per line on the same sink the aggregate document goes to. The line shape is byte-equal to the equivalent entry in the aggregate mutation.results[] / mutation.compile_errors[] array, plus a "kind" discriminator and the "version" literal:

{"version":"2","kind":"start","total":42,"meta":{"id_scheme":"2",...}}
{"version":"2","kind":"result","id":"...","function":"MyApp.Counter.add/2","status":"killed","mutator":"arith","confidence":"high",...}
{"version":"2","kind":"compile_error","id":"...","message":"..."}
{"version":"2","kind":"end","meta":{"id_scheme":"2",...},"aborted":false,"abort_reason":null,"truncated":false,"kill_rate":0.81,...}

Per-site lines ("result" / "compile_error") arrive in input order even under --max-concurrency > 1, because the runner collects via async_stream's :ordered: true default and the streaming callback fires from a sequential post-fold. The final line on the sink is the aggregate document without a kind field. Timings appear only in that document. The end line includes truncated.

Progress feedback

By default, when stderr is a TTY, a one-line-per-site progress feed is written to stderr:

[12/345] killed   lib/foo.ex:42 :arith
[13/345] survived lib/foo.ex:43 :arith
[14/345] timeout  lib/foo.ex:51 :case_drop

Pass --no-progress to suppress unconditionally. The feed is driven by the mutation runner's :on_site_completed callback (the same per-site seam --stream rides) via MutagenEx.Progress.

Resource caps

mix mutagen enforces caps on input and output volume to keep one bad invocation from running away with the host's memory or wall-clock time:

--json path safety

--json <path> is canonicalised before any mutation runs, in two layers:

  1. Parse-time (pure-string): .. segments and NUL bytes are refused with abort_reason: "unsafe_json_path". No filesystem touch happens.
  2. Filesystem-canonicalisation: every existing component is resolved through File.read_link/1. If the fully-resolved path escapes the project root (resolved through symlinks itself — macOS /var -> /private/var is handled correctly), the run aborts with abort_reason: "unsafe_json_path". The final component is allowed to not yet exist; it is created at write time.

The default policy is inside the project root only. CI workflows writing artifacts to /tmp or a sibling directory must pass --unsafe-json-outside-project; that flag bypasses the inside-root check and emits a one-shot stderr warning naming the resolved target. The symlink-resolution step still runs in either mode — the resolved path you write to is always the fully-canonical one.

Application configuration

mutagen_ex reads one optional knob from Application.get_env/3:

Key Purpose
:redact A list of compiled regexes or regex source strings. Mutagen validates it after loading runtime configuration and before the test alias and helper. Invalid entries abort with invalid_redact_config. Built-in rules run first. The default adds no custom patterns.

Set it in config/config.exs:

config :mutagen_ex,
  redact: [
    ~r/AWS_(?:ACCESS_KEY_ID|SECRET_ACCESS_KEY)=\S+/,
    ~r/Bearer [A-Za-z0-9._-]+/
  ]

The default report contains repository source and structured test metadata. Treat it as sensitive as the source. Secrets committed in source can appear in excerpts. --no-source omits site excerpts, file, function, and the fixed compiler diagnostic. Scope and test paths, coverage paths, skipped paths, module names, and site IDs still appear. Each site ID contains a path and an unkeyed MD5 hash of the normalized source AST. A reader can guess a low-entropy literal and check the hash offline.

Default reports keep failure counts by module, literal helper tag keys, and counts of lost registered processes. Compiled project exception names count as repository text. Dynamic exception names, test names, helper filter values, host process names, and diagnostic bodies require --include-host-output. Logger and cleanup output stay on stderr.

Treat an opted-in report as sensitive as the test output. Pattern redaction is best-effort. It covers known credential shapes and can miss plain values, prose, and encodings. It examines at most 64 KiB per field. A regex execution error replaces the entire field with [REDACTED: redaction failed].

Emitted text bodies and source excerpts have a 4 KiB payload cap. Source excerpts preserve tabs and newlines and remove other control characters and Unicode format characters. A truncation marker reports bytes dropped by the input and output caps.

Exit codes

Code Meaning
0 Mutagen wrote its report. This includes incomplete runs and runs where every mutation survived.
2 Mutagen aborted and wrote an error document to the selected destination. The string field abort_reason names the failure, for example "missing_scope" or "baseline_red".

A host callback that calls System.halt/1 decides the VM exit code. An after_suite callback can therefore leave a completed report with aborted: false and a non-zero exit.

Abort reasons

abort_reason Meaning
arity_required The function scope omits its arity.
baseline_red One or more tests failed against the original code.
baseline_timeout The baseline phase exceeded its deadline.
colon_syntax_unsupported A function scope uses colon syntax.
cover_already_running An existing OTP coverage session owns modules.
cover_compile_failed OTP coverage could not instrument a selected module.
cover_module_unavailable The OTP coverage module failed to load.
coverage_timeout The coverage phase exceeded its deadline.
diff_unavailable Git cannot provide a usable diff for the changed-code run.
escalation_crashed An unexpected exception ended automatic test discovery.
ex_unit_run_failed ExUnit raised, threw, or exited while running a suite.
file_not_found A selected source or test file does not exist.
file_read_failed Mutagen could not read a required file.
flag_not_supported_in_v1 An option requests an unsupported output format.
function_not_found The scoped function and arity could not be found.
host_app_start_failed The host application failed to start.
host_compile_failed Mix failed to compile the host project.
host_loadpaths_failed Mix failed to load the host project dependencies.
host_setup_timeout Host setup exceeded its deadline.
invalid_boot_timeout The host setup and cleanup timeout is invalid.
invalid_budget_ms The mutation phase time budget is invalid.
invalid_input An argument or runner input is malformed.
invalid_line The test location has an invalid line number.
invalid_max_concurrency The mutation dispatch limit is invalid.
invalid_max_sites The mutation site cap is invalid.
invalid_mix_env The command selected an unsupported Mix environment.
invalid_redact_config The configured report redaction patterns are invalid.
invalid_seed The ExUnit seed is invalid.
invalid_site The site selector has an invalid format.
invalid_since_ref --since has an invalid Git ref.
invalid_summary_max The summary limit must be a non-negative integer.
invalid_tag_name The test tag contains unsupported characters or has an invalid length.
invalid_target The test selector has an unsupported format.
invalid_timeout The per-site timeout is invalid.
missing_scope No mutation scope was supplied.
missing_tests No test selection was supplied.
module_beam_missing The compiled module could not be located.
module_not_found The scope module could not be resolved.
no_tests_match A test selector matched no files.
no_tests_run ExUnit executed no tests in coverage or baseline.
parse_error Mutagen could not parse a source or test file.
runtime_load_failed The MutagenEx runtime or a report module could not load.
self_mutation_refused The scope targets MutagenEx code in the active VM.
site_not_found The site ID is absent from the current selection.
summary_path_conflict The JSON and Markdown paths resolve to the same file.
summary_write_failed Mutagen could not write the Markdown file.
test_alias_failed A host test alias setup step failed.
test_file_load_failed A cited test file could not be loaded or matched to its compiled path.
test_helper_load_failed A required test helper could not load.
too_many_sites The scope yielded more sites than the configured cap.
too_many_targets The command exceeds the scope or test target limit.
umbrella_root The command ran from an umbrella root.
unknown_flag The command contains an unrecognised option.
unrecognised_target The scope target has an unsupported format.
unrecoverable_restore_failure Mutagen could not restore the original module.
unsafe_json_path The report path failed validation.
unsafe_summary_path The Markdown path failed validation.
unsupported_elixir_version The Elixir release is below 1.18.0.
unsupported_otp_version The Erlang/OTP release is below 27.

JSON output

The aggregate and the stream end record include summary. The examples below show the existing pipeline fields. See Reading the summary for the ranked findings and the per-function counts.

Use --json PATH for a guaranteed machine-readable report. The file ends with a newline. Output to stdout is best effort. With --stream, it follows the per-site events.

meta.timings_ms measures elapsed milliseconds for boot, coverage, baseline and mutation. The total also includes work between phases such as enumeration, through report assembly. A phase that did not run shows zero. meta.timings_ms.mutant_deadline records the effective per-mutant deadline in milliseconds. Escalation records the baseline duration and deadline for each pass that runs mutants. Captured host output can contain timestamps, process identifiers, and references.

Successful run

This example comes from a copy of the lane fixture with ExUnit.start() in test/test_helper.exs. The output is indented for readability.

MIX_ENV=test mix mutagen --scope lib/lane_fixture/arith.ex \
  --tests test/lane_fixture/arith_test.exs
{
  "abort_reason": null,
  "aborted": false,
  "baseline": {
    "failed": 0,
    "failures": [],
    "passed": 5
  },
  "coverage": {
    "covered_lines": {
      "lib/lane_fixture/arith.ex": [
        12,
        14,
        16,
        18
      ]
    }
  },
  "details": {},
  "meta": {
    "elixir_version": "1.19.6",
    "exunit_seed": 0,
    "fail_fast": true,
    "id_scheme": "2",
    "max_concurrency": 1,
    "otp_version": "28",
    "timings_ms": {
      "baseline": 2,
      "boot": 93,
      "coverage": 49,
      "mutation": 98,
      "total": 266
    },
    "tool_version": "0.2.0"
  },
  "mutation": {
    "compile_error": 0,
    "compile_errors": [],
    "completed": 7,
    "kill_rate": 0.7142857142857143,
    "killed": 5,
    "results": [
      {
        "after": "a - b",
        "before": "a + b",
        "before_source": "a + b",
        "column": 24,
        "confidence": "high",
        "file": "lib/lane_fixture/arith.ex",
        "function": "LaneFixture.Arith.add/2",
        "id": "lib/lane_fixture/arith.ex:f7fc9974f882553f6b11c5428b0baac6:arith",
        "line": 12,
        "mutator": "arith",
        "status": "killed",
        "tainted_predecessors": false,
        "warnings": []
      },
      {
        "after": "a + b",
        "before": "a - b",
        "before_source": "a - b",
        "column": 24,
        "confidence": "high",
        "file": "lib/lane_fixture/arith.ex",
        "function": "LaneFixture.Arith.sub/2",
        "id": "lib/lane_fixture/arith.ex:02f1348e12b94f92cdcfe1d5cbd6634a:arith",
        "line": 14,
        "mutator": "arith",
        "status": "killed",
        "tainted_predecessors": false,
        "warnings": []
      },
      {
        "after": "a / b",
        "before": "a * b",
        "before_source": "a * b",
        "column": 24,
        "confidence": "high",
        "file": "lib/lane_fixture/arith.ex",
        "function": "LaneFixture.Arith.mul/2",
        "id": "lib/lane_fixture/arith.ex:8d516b89cd37a53d164ad5a2afe29ed0:arith",
        "line": 16,
        "mutator": "arith",
        "status": "killed",
        "tainted_predecessors": false,
        "warnings": []
      },
      {
        "after": "div_safe(a, b)",
        "before": "div_safe(a, b) when b != 0",
        "before_source": "div_safe(a, b) when b != 0",
        "column": 22,
        "confidence": "low",
        "file": "lib/lane_fixture/arith.ex",
        "function": "LaneFixture.Arith.div_safe/2",
        "id": "lib/lane_fixture/arith.ex:50c6b355d96c53576c96a09782b8533d:guard_drop",
        "line": 18,
        "mutator": "guard_drop",
        "status": "survived",
        "tainted_predecessors": false,
        "warnings": []
      },
      {
        "after": "b == 0",
        "before": "b != 0",
        "before_source": "b != 0",
        "column": 29,
        "confidence": "high",
        "file": "lib/lane_fixture/arith.ex",
        "function": "LaneFixture.Arith.div_safe/2",
        "id": "lib/lane_fixture/arith.ex:702f2afcc0d5994b0372219de4bcc4d0:compare",
        "line": 18,
        "mutator": "compare",
        "status": "killed",
        "tainted_predecessors": false,
        "warnings": []
      },
      {
        "after": "1",
        "before": "0",
        "before_source": "0",
        "column": 29,
        "confidence": "high",
        "file": "lib/lane_fixture/arith.ex",
        "function": "LaneFixture.Arith.div_safe/2",
        "id": "lib/lane_fixture/arith.ex:d821575fe16fe3710afb032918744e6b:literal",
        "line": 18,
        "mutator": "literal",
        "status": "survived",
        "tainted_predecessors": false,
        "warnings": []
      },
      {
        "after": "a * b",
        "before": "a / b",
        "before_source": "a / b",
        "column": 41,
        "confidence": "high",
        "file": "lib/lane_fixture/arith.ex",
        "function": "LaneFixture.Arith.div_safe/2",
        "id": "lib/lane_fixture/arith.ex:2edeb721487365fa8cb5322bae381f16:arith",
        "line": 18,
        "mutator": "arith",
        "status": "killed",
        "tainted_predecessors": false,
        "warnings": []
      }
    ],
    "skipped": [],
    "state_drift_warning": {},
    "survived": 2,
    "timeout": 0,
    "total": 7
  },
  "scope": [
    {
      "file": "lib/lane_fixture/arith.ex",
      "line_range": [
        1,
        19
      ],
      "module": "LaneFixture.Arith"
    }
  ],
  "tests": {
    "exclude": [],
    "files": [
      "test/lane_fixture/arith_test.exs"
    ],
    "include": []
  },
  "truncated": false,
  "version": "2",
  "warnings": []
}

Aborted run

When the run cannot proceed — for example, --scope names a module that does not exist, aborted is true and the phase blocks that never ran are null. The default details contains a fixed hint and any available structured facts. The exit code is non-zero. The reason strings are listed in the abort reason table.

{
  "abort_reason": "module_not_found",
  "aborted": true,
  "baseline": null,
  "coverage": null,
  "details": {
    "hint": "Check the module name and that its source compiles.",
    "module": "Elixir.LaneFixture.Missing",
    "note": "Stderr may carry unredacted host output. Do not publish it as a CI artifact.",
    "step": "scope"
  },
  "meta": {
    "elixir_version": "1.19.6",
    "exunit_seed": 0,
    "fail_fast": true,
    "id_scheme": "2",
    "max_concurrency": 1,
    "otp_version": "28",
    "timings_ms": {
      "baseline": 0,
      "boot": 62,
      "coverage": 0,
      "mutation": 0,
      "total": 72
    },
    "tool_version": "0.2.0"
  },
  "mutation": null,
  "scope": [],
  "tests": null,
  "truncated": false,
  "version": "2",
  "warnings": []
}

Both examples are from CLI runs on the fixture. The schema contract is in .spec/specs/json_schema.spec.md.

Baseline guard rail

Before any mutation runs, the baseline phase runs the cited tests once against unmodified code. If any cited test fails, the pipeline aborts with reason: :baseline_red and emits the error-JSON document (aborted: true, populated baseline.failed) — mutation never starts. This catches the "your suite is already red" case so kill rates are never computed against a broken baseline.

The guard works even though the coverage → baseline → mutation phase order drains ExUnit.Server (each ExUnit.run/0 consumes the server's registered-module list) and Code.require_file/1 is one-shot per path. Both the coverage and baseline phases explicitly re-register each cited module with ExUnit.Server.add_module/2 before their own ExUnit.run/0 so baseline sees the cited modules in the registry and reports real failures. The production-condition regression test for this lives in test/mutagen_ex/baseline_red_guard_test.exs (it drives the real ExUnit.Server, not a fake), with end-to-end coverage in test/mutagen_ex/end_to_end_test.exs (:baseline_red_scenario).

An earlier development snapshot documented this as a limitation (the guard "did not trip"). That was fixed by the re-registration above; the manual mix test-first workaround the old docs suggested is no longer required.

Known limitations

Changed-code planning

Planning compares changed lines and the enclosing group's source fingerprint. An edit to a module attribute such as @limit, a defguard, a defmacro, a defdelegate target, alias, import, use options, or defstruct defaults can change runtime behavior and still plan zero runtime sites. Callers of a changed private helper keep their own fingerprints. For a deletion hunk, Mutagen considers the remaining lines next to the deletion. A deletion can plan zero sites when neither neighbor touches a runtime site.

Runtime classification uses syntax. A named def or defp outside quote qualifies even if only compile-time code calls it. An expression inside unquote still falls outside the runtime plan when it sits inside quote, even if a runtime call evaluates the expression.

Performance

The source checkout includes a benchmark harness at scripts/helper_scripts/bench_ast_perf.exs. It drives the full pipeline against scripts/helper_scripts/bench_fixtures/wrd25_200sites/ and reports wall-clock, per-site time, peak :erlang.memory/0, and the SHA-256 of the emitted NDJSON. Save a baseline with the following command. Replace --baseline with --compare to compare a fresh run to the saved file.

mix run scripts/helper_scripts/bench_ast_perf.exs --baseline /tmp/bench.json

The .25 AST/perf epic measured a 1.66× wall-clock speedup on the wrd25 fixture between commit 978a995 (pre-.25) and 78b022f (post-.25.6); see the engineering appendix in CHANGELOG.md for the full table and the follow-up triage of the gap to the spec's documented 2-4× target. The byte-identity of the mutation results (site IDs, before/after slices, kill/survive verdicts, kill_rate) is preserved across the refactor — only timing and the diagnostic warning text changed.

Specs

The behavioural contract for mutagen_ex lives in .spec/:

When the implementation and the specs disagree, fix the spec first (see .spec/AGENTS.md).

Development

mix compile --warnings-as-errors
mix test

No third-party runtime dependencies. Requires Elixir 1.18.0 or later for ExUnit case registration and Erlang/OTP 27 or later for JSON and trace sessions.

Test suites

The default mix test run excludes subprocess integration tests, the end-to-end fixture runs, and the coverage spikes.

Tag What it covers How to run
:e2e_slow Full mix mutagen pipeline against the lane fixture; takes minutes per scenario. mix test --only e2e_slow
:spike C1/C2 integration spikes under test/mutagen_ex/integration/ — ~500 cover lifecycles per default run. These are the gating decision artifact for the in-process pipeline (mutagen.decision.in_process_pipeline); they must stay runnable, just not on every default invocation. mix test --only spike
:host_setup_integration Real Postgres Sandbox and Mox through a path dependency and an installed archive. mix test --only host_setup_integration test/integration/host_setup_test.exs

mix test.integration includes the Postgres fixture. It defaults to the /tmp Unix socket, port 5432, and the current OS user. Set HOST_SETUP_DB_HOST, HOST_SETUP_DB_PORT, HOST_SETUP_DB_USER, HOST_SETUP_DB_PASSWORD, or HOST_SETUP_DB_NAME to override those settings. The fixture adds a unique suffix to its database name and drops the database after the run. Its database user needs permission to create databases.

The C2 spike accepts MUTAGEN_SPIKE_ITERATIONS=<n> to override the loop count. Default is 10; the original gating run used 100:

# Quick smoke (10 iterations)
mix test --only spike

# Full gating run (100 iterations, ~as long as the original spike)
MUTAGEN_SPIKE_ITERATIONS=100 mix test --only spike

C1 retains a fixed 100-iteration count — its loop measures restore contract fidelity across module shapes, where the cycle count is the contract.