PropWise

An AST-based analyzer for identifying property-based testing candidates in Elixir codebases.

Overview

PropWise analyzes your Elixir code to find functions that would benefit from property-based testing. It examines the Abstract Syntax Tree (AST) of your code to:

Features

Purity Analysis

Detects side effects by analyzing function calls:

Pattern Detection

Identifies functions with characteristics ideal for property testing:

Inverse Pair Detection

Finds function pairs that are inverses of each other:

Concrete Test Generation

Generates ready-to-use property-based test code:

Installation

As a Library

Add propwise to your list of dependencies in mix.exs:

def deps do

  [

    {:propwise, "~> 0.2"}

  ]

end

As a Command-Line Tool

Option 1: escript (Recommended for standalone use)

Build and install the standalone executable:

cd propwise

mix deps.get

mix escript.build



# Copy to a directory in your PATH

sudo cp propwise /usr/local/bin/

# Or just use it directly

./propwise

The escript bundles all dependencies and works without Mix or any additional setup.

Option 2: Mix archive

Install globally from Hex as a Mix archive:

mix archive.install hex propwise

This makes the mix propwise task available in any project. Note: This requires jason to be available in your Mix environment.

To uninstall:

mix archive.uninstall propwise

Option 3: As a dependency

When added as a project dependency, PropWise provides a Mix task:

mix propwise

Usage

Command Line

Using escript

# Analyze current project

./propwise .



# Analyze with custom minimum score

./propwise --min-score 5 ./my_project



# Output as JSON

./propwise --format json ./my_project



# Use PropEr instead of stream_data

./propwise --library proper ./my_project



# Show help

./propwise --help

Using Mix task

# Analyze current project

mix propwise



# Analyze with custom minimum score

mix propwise --min-score 5



# Output as JSON

mix propwise --format json



# Use PropEr instead of stream_data

mix propwise --library proper



# Analyze another project

mix propwise ../other_project



# Show help

mix propwise --help

As a Library

# Analyze a project

result = PropWise.analyze("./my_project")



# Analyze with custom options

result = PropWise.analyze("./my_project", min_score: 5, library: :proper)



# Print the report

PropWise.print_report(result)



# Print as JSON

PropWise.print_report(result, format: :json)

Analyzing a Subset of Files (for Pull Requests)

PropWise supports analyzing specific files or file subsets. This is ideal for PR workflows to focus analysis only on modified or newly added code.

  1. Single File:

    mix propwise lib/my_app/user.ex

    # or

    ./propwise lib/my_app/user.ex

  2. Comma-Separated File List (--files flag):

    mix propwise --files "lib/my_app/user.ex,lib/my_app/auth.ex"

  3. Multiple File Path Arguments:

    mix propwise lib/my_app/user.ex lib/my_app/auth.ex

CI / PR Integration

You can easily integrate PropWise into your CI pipeline to run property testing candidate detection on Pull Requests:

Bash / CI Pipeline Script

# Get modified or added .ex files relative to main branch

CHANGED_FILES=$(git diff --name-only origin/main...HEAD | grep '\.ex$' | paste -sd,)



if [ -n "$CHANGED_FILES" ]; then

  mix propwise --no-fail --files "$CHANGED_FILES"

else

  echo "No Elixir files modified in this PR."

fi

GitHub Actions Example (.github/workflows/propwise.yml)

name: PropWise PR Analysis



on:

  pull_request:

    branches: [ main ]



jobs:

  analyze-pr:

    runs-on: ubuntu-latest

    steps:

      - uses: actions/checkout@v4

        with:

          fetch-depth: 0



      - uses: erlef/setup-beam@v1

        with:

          elixir-version: '1.18'

          otp-version: '27'



      - run: mix deps.get



      - name: Run PropWise on PR Changed Files

        run: |

          CHANGED_FILES=$(git diff --name-only origin/main...HEAD | grep '\.ex$' | paste -sd,)

          if [ -n "$CHANGED_FILES" ]; then

            mix propwise --no-fail --files "$CHANGED_FILES"

          else

            echo "No Elixir files modified in this PR."

          fi

Example Output

================================================================================

PropWise Analysis Report

================================================================================



Summary:

  Total functions analyzed: 143

  Property test candidates: 24

  Candidates dropped (below threshold): 12

  Coverage: 16.8%



--------------------------------------------------------------------------------

Inverse Function Pairs Detected:

--------------------------------------------------------------------------------



  MyApp.Encoder.encode/1 <-> decode/1

  Suggestion: Test round-trip property: decode(encode(x)) == x



--------------------------------------------------------------------------------

Top Candidates (sorted by score):

--------------------------------------------------------------------------------



MyApp.Parser.parse_json/1

  Score: 6

  Location: lib/my_app/parser.ex:42

  Type: public

  Patterns:

    - Parser: Parser function

  Testing suggestions:

    - property "parse returns expected structure" do

        check all input <- string(:alphanumeric) do

          case Parser.parse_json(input) do

            {:ok, result} ->

              # TODO: Add structural assertions for parsed output.

              assert result != nil

            {:error, _} -> true

          end

        end

      end



MyApp.List.merge_sorted/2

  Score: 8

  Location: lib/my_app/list.ex:15

  Type: public

  Patterns:

    - Collection Operation: Uses Enum collection operations

    - Algebraic Structure: Potentially algebraic operation

  Testing suggestions:

    - property "associativity" do

        check all a <- term(), b <- term(), c <- term() do

          assert List.merge_sorted(List.merge_sorted(a, b), c) ==

                 List.merge_sorted(a, List.merge_sorted(b, c))

        end

      end

Scoring System

Functions are scored based on multiple factors:

Default minimum score is 4, but this can be adjusted based on your needs.

For detailed information about all detection criteria and scoring rules, see Scoring.

Configuration

You can customize PropWise's behavior by creating a .propwise.exs file in your project root.

Example Configuration

# .propwise.exs

%{

  # Directories to analyze (relative to project root)

  # Default: ["lib"]

  analyze_paths: ["lib"],



  # Property-based testing library to use for suggestions

  # Options: :stream_data (default) or :proper

  library: :stream_data



  # You can analyze multiple directories:

  # analyze_paths: ["lib", "src", "apps/my_app/lib"]

}

Configuration Options

If no .propwise.exs file is present, PropWise will use the defaults.

Options

CLI Options

Note: CLI options override configuration file settings.

Library Options

How It Works

  1. Parse: Recursively finds all .ex files in configured directories (default: lib)
  2. Extract: Parses each file's AST and extracts function definitions
  3. Analyze Purity: Walks the AST to detect side effects
  4. Detect Patterns: Looks for common patterns in function structure and naming
  5. Score: Calculates a testability score for each function
  6. Find Pairs: Identifies inverse function pairs across the codebase
  7. Generate Suggestions: Creates concrete property-based test examples using your chosen library
  8. Report: Presents findings with ready-to-use test code

Limitations

Security Note

PropWise loads configuration from .propwise.exs files using Code.eval_file/1, which executes arbitrary Elixir code. Only analyze projects you trust.

Contributing

Contributions are welcome! Areas for improvement:

License

MIT