exavier

A mutation testing library in Elixir. Inspired by pitest (http://pitest.org) and mutant.

What is Mutation Testing?

Mutation testing [...] is used to design new software tests and evaluate the quality of existing software tests. Mutation testing involves modifying a program in small ways. Each mutated version is called a mutant and tests detect and reject mutants by causing the behavior of the original version to differ from the mutant. This is called killing the mutant. Test suites are measured by the percentage of mutants that they kill. New tests can be designed to kill additional mutants.

- Wikipedia

How does `exavier` work?

The exavier library mutates code in parallel per module, but mutates each module sequentially per mutator. Initial code line coverage analysis is done sequentially for all modules as a pre-processing step. It is better explained as follows:

Run code line coverage analysis for each module, sequentially
Mutate the code according to each available mutator
1. For each module, in parallel:
  1. For each mutator, sequentially:
    1. Mutate code with given mutator
    2. Run tests once again (now against mutated code)
    3. Record results (% mutants survived vs. killed)

Mutators

Mutators specify ways in which we can mutate the code. Currently we have 13 mutators available in exavier:

AOR stands for "Arithmetic Operator Replacement". There are several possibilities for replacing an arithmetic operator. We follow the ones defined by pitest. Similarly, ROR stands for "Relational Operator Replacement". IfTrue is inspired by pitest's "Remove Conditionals". NegateConditionals is also inspired by pitest.

You can create new mutators. You just have to make sure they abide to the interface provided by behaviour Exavier.Mutators.Mutator:

defmodule Exavier.Mutators.Mutator do
  @type operator() :: atom()
  @type metadata() :: keyword()
  @type args() :: term()

  @type ast_node() :: {operator(), metadata(), args()}
  @type lines_to_mutate() :: [integer()]

  @callback operators() :: [operator()]
  @callback mutate(ast_node(), lines_to_mutate()) :: ast_node() | :skip
end

An Exavier.Mutators.Mutator has two mandatory functions:

operators/0
1. input:
  - (none)
2. output:
  - an array of atoms (operators to which the mutation can be applied, e.g., [:==, :>=])
mutate/2
1. input:
  - AST node (e.g., {operator, meta, args})
  - lines that should be mutated as part of that mutation (array of integers, e.g., [3, 6])
2. output:
  - mutated AST node (e.g., {operator, meta, args})

Add custom mutators to .exavier.exs under the :custom_mutators key:

%{
  ...
  custom_mutators: [
    MyApp.MySpecialMutator
  ]
  ...
}

Installation

The package can be installed by adding exavier to your list of dependencies in mix.exs:

def deps do
  [
    {:exavier, "~> 0.3.0"}
  ]
end

Run mix exavier.test and you should see output similar to this:

......................

(...)

16) test when infinity (Elixir.HelloWorldTest)
  - if(y == :special) do
  -   :yes
  - else
  -   :no
  - end

  + if(true) do
  +   :yes
  + else
  +   :no
  + end

/Users/dnlserrano/Repos/exavier/test/hello_world_test.exs:10


22 tests, 6 failed (mutants killed), 16 passed (mutants survived)
27.27% mutation coverage

Options

exavier provides the following configuration options via dotfile .exavier.exs:

:threshold: Mutation testing coverage threshold (can be integer or floating point number)

# .exavier.exs

%{
  ...
  threshold: 67
  ...
}

:test_files_to_modules: Overrides the default mapping of finding a module based on its test file name. E.g., test file test/my_file_abc_test.exs might be testing module MyFileABC instead of MyFileAbc (exavier's default mapping)

# .exavier.exs

%{
  ...
  test_files_to_modules: %{
    "test/my_file_abc_test.exs" => MyFileABC
  }
  ...
}

:custom_mutators: Adds mutator modules to be run during tests. See the Mutators section for directions on how to create your own mutators.

%{
  ...
  custom_mutators: [
    MyApp.MySpecialMutator
  ]
  ...
}

To be done

This is for now just a proof-of-concept. A lot of it has been no more than a joyful exercise in exploring what tools Erlang and Elixir provide to make such a library possible. Among some things I'd love to tackle in the near future are:

Add way more tests (OMG the irony, forgive me, this is still a bit of a PoC as you can tell by the length of this "To be done" section)
Add mutators
- [x] AOR1
- [x] AOR2
- [x] AOR3
- [x] AOR4
- [x] ROR1
- [x] ROR2
- [x] ROR3
- [x] ROR4
- [x] ROR5
- [x] Remove Conditionals
  - Can still be done for case, unless
- [x] Conditionals Boundary
- [x] Negate Conditionals
- [x] Invert Negatives
Ability to tune which mutators are used
Ability to add custom mutators defined by the user (i.e., not in exavier)
Analyse if we really should or shouldn't care about pre-processing step of code line coverage
Have other ways of terminating mutation test suite (e.g., fast-fail if threshold of X mutants have survived)
Parallelise mutating module per mutator

More info

Discussion of the library at ElixirForum.com
I wrote about exavier in my personal blog
Always happy to chat in the elixir-lang Slack channel over at #exavier

Library name

Inspired by Dr. Charles Xavier (Professor X) from the X-Men mutants comic books I read as a kid.

Acknowledgements

Thanks to Tita Moreira :heart: for putting up with my nerdiness.

Thanks to Richard A. DeMillo, Richard J. Lipton and Fred G. Sayward for their seminal work on Mutation Testing back in 1978, with the paper "Hints on Test Data Selection: Help for the Practicing Programmer".

Thanks to Henry Coles for pitest and Markus Schirp for mutant, which served as an inspiration for this project.

License

Copyright © 2019-present Daniel Serrano <danieljdserrano at protonmail>

This work is free. You can redistribute it and/or modify it under the
terms of the MIT License. See the LICENSE file for more details.

Made in Portugal :portugal: by dnlserrano