Muex

Mutation testing library for Elixir, Erlang, and other languages.

Muex evaluates test suite quality by introducing deliberate bugs (mutations) into code and verifying that tests catch them. It provides a language-agnostic architecture with dependency injection, making it easy to extend support to new languages.

Features

Installation

Muex can be used in three ways:

1. As a Mix Dependency (Recommended for CI/CD)

Add muex to your list of dependencies in mix.exs:

def deps do
  [
    {:muex, "~> 0.2.0", only: [:dev, :test], runtime: false}
  ]
end

Then run:

mix deps.get
mix muex

2. As a Hex Archive (Recommended for Global Use)

Install globally to use across all your projects:

mix archive.install hex muex

This makes mix muex available in any Elixir project without adding it as a dependency.

3. As an Escript (Standalone Binary)

For standalone usage or distribution:

# From muex repository
mix escript.build

# Install system-wide
sudo cp muex /usr/local/bin/

# Use in any project
cd /path/to/your/project
muex

For detailed installation instructions and comparison, see docs/INSTALLATION.md.

Usage

Run mutation testing on your project:

# Using mix task (dependency or hex archive)
mix muex

# Using escript (standalone binary)
muex

Both commands accept the same options and produce identical results.

By default, Muex intelligently filters files to focus on business logic and skip framework code. This dramatically reduces the number of mutations tested.

File Filtering Options

# Use intelligent filtering (default)
mix muex

# Show which files are included/excluded
mix muex --verbose

# Adjust minimum complexity score (default: 20)
mix muex --min-score 30

# Disable filtering to test all files
mix muex --no-filter

# Limit total mutations tested
mix muex --max-mutations 500

File Selection

# Run on specific directory
mix muex --files "lib/myapp"

# Run on specific file
mix muex --files "lib/my_module.ex"

# Run on files matching glob pattern (single level)
mix muex --files "lib/muex/*.ex"

# Run on files matching recursive glob pattern
mix muex --files "lib/**/compiler*.ex"
mix muex --files "lib/{muex,mix}/**/*.ex"

Other Options

# Use specific mutators
mix muex --mutators arithmetic,comparison,boolean

# Set concurrency and timeout
mix muex --concurrency 4 --timeout 10000

# Fail if mutation score below threshold
mix muex --fail-at 80

# Enable mutation optimization (balanced preset)
mix muex --optimize

# Use conservative optimization (best balance)
mix muex --optimize --optimize-level conservative

# Use aggressive optimization (fastest)
mix muex --optimize --optimize-level aggressive

# Custom optimization settings
mix muex --optimize --min-complexity 3 --max-per-function 15

Mutation Optimization

Muex includes sophisticated heuristics to reduce the number of mutants tested while maintaining mutation testing effectiveness. This can reduce testing time by 50-70% with minimal impact on mutation score.

When to Use Optimization

Optimization Levels

Conservative (recommended for CI/CD):

mix muex --optimize --optimize-level conservative

Balanced (default, good for development):

mix muex --optimize

Aggressive (rapid checks only):

mix muex --optimize --optimize-level aggressive

How It Works

The optimizer uses 7 strategies:

  1. Equivalent Mutant Detection: Filters semantically equivalent mutations
  2. Code Complexity Scoring: Skips mutations in trivial code (getters, simple guards)
  3. Impact Scoring: Prioritizes mutations by risk level (1-11 points)
  4. Mutation Clustering: Groups similar mutations and samples representatives
  5. Per-Function Limits: Caps mutations per function to prevent explosion
  6. Boundary Prioritization: Always preserves critical comparison mutations
  7. Pattern-Based Filtering: Removes known low-value mutations

For detailed information, see docs/MUTATION_OPTIMIZATION.md.

Example: Cart Project

Real-world results from the shopping cart example (440 LOC, 84 tests):

Mode Mutations Time Score Best For
Baseline 886 ~3 min 99.77% Final validation
Conservative 308 ~1 min 99.35% CI/CD
Balanced 28 ~10 sec 89.29% Development

See examples/cart/OPTIMIZATION_RESULTS.md for complete analysis.

Compile-Time Configuration

Custom language adapters and mutators can be registered in your config/config.exs so they are available via the CLI flags:

# config/config.exs
import Config

# Register a custom language adapter (usable with --language lua)
config :muex, languages: %{"lua" => MyApp.Language.Lua}

# Register custom mutators (usable with --mutators string,regex)
config :muex, mutators: %{
  "string" => MyApp.Mutator.String,
  "regex"  => MyApp.Mutator.Regex
}

Language adapter modules must implement the Muex.Language behaviour and mutator modules must implement the Muex.Mutator behaviour. Custom entries are merged with the built-in ones at compile time; entries with the same key override the built-in default.

Available Mutators

Muex provides 6 comprehensive mutation strategies:

Supported Languages

Both languages benefit from hot module swapping for efficient mutation testing.

Example Output

Loading files from lib...
Found 24 file(s)
Analyzing files for mutation testing suitability...
Selected 8 file(s) for mutation testing
Skipped 16 file(s) (low complexity or framework code)
Generating mutations...
Testing 342 mutation(s)
Analyzing test dependencies...
Running tests...

Mutation Testing Results
==================================================
Total mutants: 342
Killed: 287 (caught by tests)
Survived: 55 (not caught by tests)
Invalid: 0 (compilation errors)
Timeout: 0
==================================================
Mutation Score: 83.9%

With --verbose flag:

Loading files from lib...
Found 24 file(s)
Analyzing files for mutation testing suitability...
  ✗ lib/mix/tasks/muex.ex (Mix task)
  ✗ lib/muex/application.ex (Application/Supervisor)
  ✓ lib/muex/compiler.ex (score: 91)
  ✓ lib/muex/runner.ex (score: 83)
  ✗ lib/muex/language.ex (Behaviour definition)
  ...

Output Formats

Terminal (Default)

Colored terminal output with progress indicators and summary:

Mutation Testing Results
==================================================
Total mutants: 25
Killed: 20 (caught by tests)
Survived: 5 (not caught by tests)
Invalid: 0 (compilation errors)
Timeout: 0
==================================================
Mutation Score: 80.0%

JSON Format

Structured JSON for CI/CD integration:

mix muex --format json
# Outputs: muex-report.json

HTML Format

Interactive HTML report with color-coded results:

mix muex --format html
# Outputs: muex-report.html

Examples

See the examples/ directory for example projects:

Note: The examples demonstrate the mutation testing concept. For production use, consider integrating Muex into your project's mix.exs as a dependency.

Documentation

Documentation can be found at https://hexdocs.pm/muex.