rebar3_mutate

Build Status Hex.pm Docs Slack

Mutation testing plugin for rebar3, the Erlang counterpart to PIT (Java), Stryker (JavaScript, .NET) and mutmut (Python). Systematically applies small code transformations (mutants) and runs your tests to verify they catch them.

Installation

Add to project_plugins in your rebar.config:

{project_plugins, [rebar3_mutate]}.

Usage

Run against all modules:

rebar3 mutate

Target specific modules:

rebar3 mutate -m my_module,my_other_module

Exclude modules:

rebar3 mutate -x generated_module,test_helper

Select specific operators:

rebar3 mutate -o op_arithmetic,op_boolean

Set a per-mutant timeout:

rebar3 mutate -t 10000

Use Common Test instead of EUnit:

rebar3 mutate -f ct

Each module runs <module>_SUITE by default. Override it with --suite:

rebar3 mutate -f ct --suite my_integration_SUITE

Enforce a minimum score in CI:

rebar3 mutate -s 80

Output JSON for CI integration:

rebar3 mutate --format json

Control parallelism:

rebar3 mutate -w 4

Only mutate lines changed since a base ref (fast enough to run on every PR):

rebar3 mutate --diff origin/main

In CI

Taure/erlang-ci ships a ready-made action:

- uses: Taure/erlang-ci/mutate@v2.1.1
  with:
    min-score: '80'
    diff: 'true'
    diff-base: 'origin/main'

Options

Flag Short Description Default
--module -m Target module(s), comma-separated all
--exclude -x Modules to exclude, comma-separated none
--operators -o Mutation operators to use, comma-separated all
--timeout -t Per-mutant timeout in milliseconds 5000
--test-framework -f Test framework: eunit or ct eunit
--suite Common Test suite to run <module>_SUITE
--min-score -s Minimum mutation score (0-100), fail if below none
--format Output format: console or json console
--workers -w Workers used to compile mutants in parallel scheduler count
--diff -d Only mutate lines changed since a base ref (e.g. origin/main) none

--workers parallelises mutant compilation only. Mutants are loaded and tested one at a time, because the code server holds at most two versions of a module.

Configuration

Options that belong to the project rather than to one run go in a mutate section of rebar.config.

Extra EUnit test modules

By default a module is scored against its own tests only: <module>_tests and any test functions it exports. A module whose behaviour is exercised by integration test modules then scores close to 0% although those suites kill its mutants. Map it to the test modules that count for it:

{mutate, [
    {eunit_tests, #{
        kura_repo_worker => [kura_integration_assoc_tests],
        kura_preloader => [kura_integration_assoc_tests, kura_sandbox_tests]
    }}
]}.

For a mapped module the baseline and every mutant run eunit:test([Module | ExtraTestModules], [no_tty]), so the module's own tests still count. Unmapped modules behave as before, apart from the repeatability check in Baseline. The mapping only applies with -f eunit; with -f ct it is ignored with a warning.

The config is validated before anything runs. Each of these stops the run with an error rather than being skipped, so a green score always means the listed tests ran:

Mutants are tested one at a time. The mapped suites must also be repeatable, each run cleaning up after itself; the plugin checks that (see Baseline) and fails the module if not.

Mutation Operators

Operator Mutations
op_arithmetic + <-> -, * <-> div, rem -> div
op_relational > <-> <, >= <-> =<, =:= <-> =/=, == <-> /=
op_boolean andalso <-> orelse, true <-> false
op_return_value ok <-> error atoms and tuples
op_statement_delete Replace function calls with ok
op_constant Integer N -> N+1, N-1, 0
op_negate_condition Wrap andalso/orelse with not, remove existing not
op_list ++ <-> --, hd <-> tl (including erlang:hd/1 and erlang:tl/1)

Only function bodies defined in the module's own source are mutated. Attributes carry no runtime behaviour, so mutating a -spec type atom or an -export arity produces a mutant that is either inert or a guaranteed compile error. Test functions (*_test/0, *_test_/0) and anything pulled in from an include are skipped for the same reason. The scaffolding of logger macro calls (?LOG_* and ?LOG from kernel/include/logger.hrl) is skipped: the expansion's level, line and location constants and its logger:allow/2 case are not something a test can observe. The arguments you pass are mutated as usual, so a call or expression inside a log report, or a level given as an expression, still produces mutants.

Baseline

Before mutating a module, the plugin runs its tests unmutated:

Timeouts

A mutant whose tests run past --timeout is recorded as timed out, and its test run is stopped before the next mutant loads. Each EUnit and Common Test run is given a group leader of its own, and every EUnit run its own EUnit server, so test processes and anything they spawn inherit that group leader, directly or through EUnit's per-test group leaders. When the run ends, the plugin kills every process started during the run whose group leader chain leads back to the run, or ends at a group leader that was started during the run and has since exited (EUnit stops a test's group leader when the test ends, leaving unlinked helpers pointing at it), and waits for them to exit. Common Test runs go through the same sweep. A test blocked in its own code (for example waiting on the database) is stopped before the next mutant's run. The sweep is best effort: a process in test or dependency code that keeps replacing itself with a new process can stay ahead of it. A process outside the run that is linked to a process the sweep kills, and does not trap exits, is taken down with it. Processes a test started under an application are not part of the run and are left alone, except that restoring the original module kills any process still running mutant code.

Progress Indicator

During execution, a per-mutant progress indicator is printed to stderr:

Interpreting Results

The plugin reports:

The mutation score is killed / (killed + survived + timed out). Compile errors and skipped mutants never ran a test, so they are excluded from the denominator. This is the same number --min-score gates on.

Known Limitations

The plugin cannot mutate itself. It swaps modules in the one code server it shares with the code under test, so a mutant of one of its own modules would replace code the run is executing; restoring the original then has to purge a version a live process is using, which kills that process. Its own modules are reported as skipped with the reason self_mutation.

If your EUnit tests live outside the <module>_tests.erl convention (for example one central myapp_tests.erl for the whole app), the plugin cannot associate them with the module under test. Such modules are reported as skipped rather than scored, so the number stays honest, but they are not covered. Tracked in #13; coverage-guided test selection is the fix. Colocated <module>_tests.erl files and Common Test suites are unaffected.

License

MIT -- see LICENSE.