ExUnitBuildkite

Real-time Buildkite build annotations for ExUnit test failures. The Elixir equivalent of rspec-buildkite.

Before: Test failures are buried in log output. You find out what failed after scrolling through hundreds of lines, or you add a separate junit-annotate pipeline step that runs after all tests complete.

After: Each failure is annotated on the build the instant it happens — while the rest of the suite is still running. No extra pipeline step, no JUnit XML, no post-processing.

┌─────────────────────────────────────────────────────────────────┐
│ 🔴 Test Failures                                   ── Build #42│
│                                                                 │
│ ▸ creates a customer with valid params                          │
│   — MyApp.CustomersTest (test/customers_test.exs:15)            │
│                                                                 │
│ ▸ rejects duplicate email addresses                             │
│   — MyApp.CustomersTest (test/customers_test.exs:42)            │
│                                                                 │
│   Click to expand each failure for the full assertion output.   │
└─────────────────────────────────────────────────────────────────┘

Installation

Add exunit_buildkite to your test dependencies in mix.exs:

def deps do
  [
    {:exunit_buildkite, "~> 0.2", only: :test}
  ]
end

Then fetch:

mix deps.get

Quick Start

Add the formatter to test/test_helper.exs alongside the built-in CLI formatter:

formatters =
  if System.get_env("CI") do
    [ExUnitBuildkite.Formatter, ExUnit.CLIFormatter]
  else
    [ExUnit.CLIFormatter]
  end

ExUnit.start(formatters: formatters)

That's it. Push to Buildkite and your next test failure will appear as a build annotation in real-time.

Safe outside Buildkite — if buildkite-agent isn't on $PATH (local dev, GitHub Actions, etc.), the formatter silently does nothing. No errors, no overhead.

Configuration

Defaults work out of the box. Optionally configure via application env:

# config/test.exs
config :exunit_buildkite,
  context: "exunit",
  style: "error"

Options can also be passed inline to the formatter — inline values take precedence over application config:

ExUnit.start(formatters: [
  {ExUnitBuildkite.Formatter, context: "backend-tests", style: "error"},
  ExUnit.CLIFormatter
])

Monorepo Example

In a monorepo with multiple test suites, use context to keep annotations separate:

# apps/api/test/test_helper.exs
ExUnit.start(formatters: [
  {ExUnitBuildkite.Formatter, context: "api-tests"},
  ExUnit.CLIFormatter
])

# apps/worker/test/test_helper.exs
ExUnit.start(formatters: [
  {ExUnitBuildkite.Formatter, context: "worker-tests"},
  ExUnit.CLIFormatter
])

Each app's failures will appear in their own annotation block on the build page.

How It Works

ExUnitBuildkite.Formatter is a GenServer that plugs into ExUnit's formatter system. It receives the same events as the built-in CLIFormatter:

ExUnit runs tests
    │
    ├─ test passes → ignored
    │
    └─ test fails
         │
         ├─ Format failure with ExUnit.Formatter.format_test_failure/5
         │  (same output you see in the terminal)
         │
         ├─ Wrap in collapsible HTML <details> with test name + file:line
         │
         └─ Shell out to: buildkite-agent annotate --append --style error
            (appears on the build page immediately)

The formatter runs in its own process — it never blocks test execution. At the end of the suite, ExUnit waits for all formatters to drain, so every failure is guaranteed to be annotated before the step exits.

Annotation Format

Each failure produces a collapsible <details> block:

<details>
  <summary>
    <code>creates a customer with valid params</code>
    — MyApp.CustomersTest (<code>test/customers_test.exs:15</code>)
  </summary>
  <pre>
    1) creates a customer with valid params (MyApp.CustomersTest)
       test/customers_test.exs:15

       Assertion with == failed
       code:  assert customer.name == "Alice"
       left:  "Bob"
       right: "Alice"
  </pre>
</details>

Migrating from junit-annotate

If you're using junit_formatter + the junit-annotate Buildkite plugin, you can replace both with this single formatter.

What to remove

1. Remove junit_formatter from mix.exs:

- {:junit_formatter, "~> 3.4", only: :test},
+ {:exunit_buildkite, "~> 0.2", only: :test},

2. Remove junit_formatter config from config/test.exs:

- config :junit_formatter,
-   report_dir: "tmp",
-   automatic_create_dir?: true,
-   print_report_file: true,
-   include_filename?: true

3. Replace the formatter in test/test_helper.exs:

  formatters =
    if System.get_env("CI") do
-     [JUnitFormatter, ExUnit.CLIFormatter]
+     [ExUnitBuildkite.Formatter, ExUnit.CLIFormatter]
    else
      [ExUnit.CLIFormatter]
    end

4. Remove the JUnit XML upload from your test script:

  mix test
-
- if [[ -f tmp/test-junit-report.xml ]]; then
-   buildkite-agent artifact upload tmp/test-junit-report.xml
- fi

5. Remove the annotation pipeline step:

If you have a separate junit-annotate step in your pipeline (YAML or DSL), remove it entirely. Annotations now happen inline during the test step.

What you gain

Requirements

• Elixir~> 1.17
• buildkite-agent on $PATH in CI (standard on all Buildkite agents)

No other dependencies. The package depends only on ExUnit (part of Elixir) and uses buildkite-agent via System.cmd/3.

Contributing

1. Fork the repo
2. Create a feature branch (git checkout -b my-feature)
3. Make your changes
4. Run mix test && mix format --check-formatted
5. Open a pull request

Releasing

Maintainers: use bin/release <major|minor|patch> to bump, tag, and publish. Requires HEX_API_KEY env var for Hex publishing.

License

MIT — see LICENSE.