rebar3_mutate

Mutation testing plugin for rebar3. Systematically applies small code transformations (mutants) and runs your tests to verify they catch them.

Installation

Add to project_plugins in your rebar.config:

{project_plugins, [rebar3_mutate]}.

Usage

Run against all modules:

rebar3 mutate

Target specific modules:

rebar3 mutate -m my_module,my_other_module

Exclude modules:

rebar3 mutate -x generated_module,test_helper

Select specific operators:

rebar3 mutate -o op_arithmetic,op_boolean

Set a per-mutant timeout:

rebar3 mutate -t 10000

Use Common Test instead of EUnit:

rebar3 mutate -f ct

Each module runs <module>_SUITE by default. Override it with --suite:

rebar3 mutate -f ct --suite my_integration_SUITE

Enforce a minimum score in CI:

rebar3 mutate -s 80

Output JSON for CI integration:

rebar3 mutate --format json

Control parallelism:

rebar3 mutate -w 4

Only mutate lines changed since a base ref (fast enough to run on every PR):

rebar3 mutate --diff origin/main

In CI

Taure/erlang-ci ships a ready-made action:

- uses: Taure/erlang-ci/mutate@v2.1.1
  with:
    min-score: '80'
    diff: 'true'
    diff-base: 'origin/main'

Options

Flag	Short	Description	Default
`--module`	`-m`	Target module(s), comma-separated	all
`--exclude`	`-x`	Modules to exclude, comma-separated	none
`--operators`	`-o`	Mutation operators to use, comma-separated	all
`--timeout`	`-t`	Per-mutant timeout in milliseconds	5000
`--test-framework`	`-f`	Test framework: `eunit` or `ct`	eunit
`--suite`		Common Test suite to run	`<module>_SUITE`
`--min-score`	`-s`	Minimum mutation score (0-100), fail if below	none
`--format`		Output format: `console` or `json`	console
`--workers`	`-w`	Workers used to compile mutants in parallel	scheduler count
`--diff`	`-d`	Only mutate lines changed since a base ref (e.g. `origin/main`)	none

--workers parallelises mutant compilation only. Mutants are loaded and tested one at a time, because the code server holds at most two versions of a module.

Mutation Operators

Operator	Mutations
`op_arithmetic`	`+` <-> `-`, `*` <-> `div`, `rem` -> `div`
`op_relational`	`>` <-> `<`, `>=` <-> `=<`, `=:=` <-> `=/=`, `==` <-> `/=`
`op_boolean`	`andalso` <-> `orelse`, `true` <-> `false`
`op_return_value`	`ok` <-> `error` atoms and tuples
`op_statement_delete`	Replace function calls with `ok`
`op_constant`	Integer N -> N+1, N-1, 0
`op_negate_condition`	Wrap `andalso`/`orelse` with `not`, remove existing `not`
`op_list`	`++` <-> `--`, `hd` <-> `tl` (including `erlang:hd/1` and `erlang:tl/1`)

Only function bodies defined in the module's own source are mutated. Attributes carry no runtime behaviour, so mutating a -spec type atom or an -export arity produces a mutant that is either inert or a guaranteed compile error. Test functions (*_test/0, *_test_/0) and anything pulled in from an include are skipped for the same reason.

Baseline

Before mutating a module, the plugin runs its tests unmutated:

green -- proceed, and use the measured suite time to sanity-check --timeout
red -- fail the run; a mutation score computed against a failing suite is meaningless, because every mutant reads as killed
no tests -- skip the module and report it as skipped rather than scoring it 0%

Progress Indicator

During execution, a per-mutant progress indicator is printed to stderr:

. -- killed
S -- survived
T -- timed out
E -- compile error
- -- skipped

Interpreting Results

The plugin reports:

Killed -- tests caught the mutation (good)
Survived -- tests did not catch the mutation (indicates a gap)
Timed out -- tests hung on the mutation
Compile error -- mutation produced invalid code
Skipped -- the mutant could not be evaluated (for example a worker crashed)

The mutation score is killed / (killed + survived + timed out). Compile errors and skipped mutants never ran a test, so they are excluded from the denominator. This is the same number --min-score gates on.

Known Limitations

The plugin cannot mutate itself. It swaps modules in the one code server it shares with the code under test, so a mutant of one of its own modules would replace code the run is executing; restoring the original then has to purge a version a live process is using, which kills that process. Its own modules are reported as skipped with the reason self_mutation.

If your EUnit tests live outside the <module>_tests.erl convention (for example one central myapp_tests.erl for the whole app), the plugin cannot associate them with the module under test. Such modules are reported as skipped rather than scored, so the number stays honest, but they are not covered. Tracked in #13; coverage-guided test selection is the fix. Colocated <module>_tests.erl files and Common Test suites are unaffected.

License

MIT -- see LICENSE.