ReadmeTester

A library for testing Elixir code blocks in markdown files. Ensures your documentation stays in sync with your actual code.

Installation

Add readme_tester to your list of dependencies in mix.exs:

def deps do
  [
    {:readme_tester, "~> 0.1.0", only: :test}
  ]
end

Quick Start

Create a test file in test/readme_test.exs:

defmodule MyApp.ReadmeTest do
  use ReadmeTester.Case,
    files: ["README.md"],
    base_path: File.cwd!()
end

Run with mix test.

Configuration Options

Markdown Annotations

Control test behavior with HTML comments placed immediately before a code block:

Examples

Skip a block:

```elixir
def deps do
  [{:my_app, "~> 1.0"}]
end
```

Syntax check only (no execution):

```elixir
def future_feature do
  # This compiles but may not run without dependencies
  SomeModule.call()
end
```

Expect an exception:

```elixir
raise "This should raise!"
```

Verify formatting:

```elixir
def well_formatted do
  :ok
end
```

Share state between blocks:

```elixir
user = %{name: "Alice", age: 30}
```

```elixir
# This block can use `user` from above
user.name
```

IEx Blocks with Output Matching

Blocks marked with iex support output verification. Expected output can be specified in two ways:

Traditional IEx format (output on next line):

```iex
iex> 1 + 1
2
iex> [1, 2, 3] |> Enum.sum()
6
```

Inline expectation (using # =>):

```iex
iex> 1 + 1  # => 2
iex> :hello  # => :hello
```

The test fails if the actual output doesn't match the expected value. Complex values like lists, maps, and tuples are compared structurally.

Example Configuration

defmodule MyApp.DocsTest do
  use ReadmeTester.Case,
    files: ["README.md", "docs/**/*.md"],
    base_path: File.cwd!(),
    skip_patterns: [
      ~r/def deps do/,      # skip deps config
      ~r/config :/          # skip config snippets
    ],
    setup: "alias MyApp.{User, Order}"
end

Programmatic Usage

You can also use ReadmeTester programmatically:

Parsing

# Parse a markdown file - returns {:ok, blocks} or {:error, reason}
{:ok, blocks} = ReadmeTester.parse_file("README.md")

# Parse markdown content directly
blocks = ReadmeTester.parse_content("```elixir\n1 + 1\n```")

Each block is a map with the following structure:

%{
  code: "1 + 1",              # The code content
  line: 5,                     # Starting line number in the file
  language: "elixir",          # "elixir" or "iex"
  annotations: [:skip]         # List of annotations (e.g., :skip, :setup)
}

Execution

# Execute a single block
case ReadmeTester.execute(block, setup: "alias MyApp.Helper") do
  {:ok, result} -> IO.puts("Returned: #{inspect(result)}")
  {:ok, result, bindings} -> IO.puts("Shared: #{inspect(bindings)}")
  {:skipped, :annotated} -> IO.puts("Block was skipped")
  {:error, :compile_error, message} -> IO.puts("Compile error: #{message}")
  {:error, :runtime_error, exception, stacktrace} -> IO.puts("Runtime error")
  {:error, :output_mismatch, %{expected: e, actual: a}} -> IO.puts("Expected #{e}, got #{a}")
  {:error, :expected_exception, message} -> IO.puts("Should have raised")
  {:error, :format_error, message} -> IO.puts("Not formatted: #{message}")
end

# Execute multiple blocks in sequence (with shared state)
results = ReadmeTester.execute_all(blocks, binding: [x: 1])
# => [{%{code: ..., line: ...}, {:ok, result}}, ...]

Testing Files

# Test all files and get a summary
result = ReadmeTester.test_files(
  ["README.md", "docs/**/*.md"],
  base_path: "/path/to/project"
)
# => %{
#      total: 10,
#      passed: 8,
#      failed: 1,
#      skipped: 1,
#      failures: [
#        {"/path/to/README.md", 42, {:compile_error, "undefined function foo/0"}},
#        ...
#      ]
#    }

How It Works

1. Parse: Extracts all elixir and iex fenced code blocks from markdown files
2. Execute: Runs each block through Code.eval_string/3
3. Report: Generates ExUnit test cases or returns a summary

Code blocks are executed in isolation. Each block is a separate test - if one fails, others still run.

Examples

Simple expressions work out of the box:

list = [1, 2, 3]
Enum.sum(list)

Module definitions are supported:

defmodule Example do
  def greet(name), do: "Hello, #{name}!"
end

Example.greet("World")

Tips for Testable Documentation

1. Self-contained examples: Write code blocks that can run independently <!-- readme_tester: skip -->for config, partial code, etc. 3. **Use the:setupoption for imports**: Put common aliases/imports in the:setup` option 4. Avoid external dependencies: Mock or skip blocks that need databases, APIs, etc.