Benchfella

Microbenchmarking tool for Elixir.

Overview

With Benchfella you can define small tests and it will intelligently run each individual one to obtain a more or less reliable estimate of the average run time for each test.

The key features of Benchfella:

• very easy to write and run microbenchmarks
• compare measurements between multiple runs
• plot results of multiple runs

If you are looking for a more elaborate treatment of the measurements, take a look at bmark which employs mathematical statistics to compare benchmarking results and determine their credibility.

Installation

Choose how you'd like to install the custom Mix tasks:

1. As an archive:

   mix archive.install https://github.com/alco/benchfella/releases/download/v0.2.1/benchfella-0.2.1.ez

   This will make the custom tasks available to mix regardless of where it is invoked, just like the builtin tasks are.

   Caveat: the archive may not always keep up with the development on the master branch.

2. Add benchfella as a dependency to your project:

   # in your mix.exs
   
   defp deps do
     [{:benchfella, "~> 0.2.0"}]
   end

   This will make the new tasks available only in the root directory of your Mix project.

Usage

Take a moment to study the output of running mix help bench and mix help bench.cmp to see all supported options.

In order to start writing tests, add a directory called bench and put files with names that match the pattern *_bench.exs in it. Then run mix bench.

Example:

# bench/basic_bench.exs
defmodule BasicBench do
  use Benchfella

  @list Enum.to_list(1..1000)

  bench "hello list" do
    Enum.reverse @list
  end
end

$ mix bench
Settings:
  duration:      1.0 s

## BasicBench
[13:23:58] 0/1: hello list

Finished in 3.15 seconds

## BasicBench
hello list      500000   5.14 µs/op

setup_all and teardown_all

setup_all/0 lets you perform some code before the first test in a module is run. It takes no arguments and should return {:ok, <context>} where <context> is any term, it will be passed into before_each_bench/1 and teardown_all/1 if they are defined. Returning any other value will raise an error and cause the whole module to be skipped.

teardown_all/1 lets you do some cleanup after the last test in a module has finished running. It takes the context returned from setup_all/0 (nil by default) as its argument.

# bench/sys_bench.exs
defmodule SysBench do
  use Benchfella

  setup_all do
    depth = :erlang.system_flag(:backtrace_depth, 100)
    {:ok, depth}
  end

  teardown_all depth do
    :erlang.system_flag(:backtrace_depth, depth)
  end

  @list Enum.to_list(1..10000)

  bench "list reverse" do
    Enum.reverse(@list)
  end
end

before_each_bench and after_each_bench

before_each_bench/1 runs before each individual test is executed. It takes the context returned from setup_all/0 and should return {:ok, <bench_context>} where <bench_context> is any term. Returning any other value will raise an error and cause the current test to be skipped.

<bench_context> returned from before_each_bench/1 will be available as the bench_context variable in each test.

after_each_bench/1 runs after each individual test has been executed. It takes the context returned from before_each_bench/1 as its argument.

# bench/ets_bench.exs
defmodule ETSBench do
  use Benchfella

  before_each_bench _ do
    tid = :ets.new(:my_table, [:public])
    {:ok, tid}
  end

  after_each_bench tid do
    IO.inspect length(:ets.tab2list(tid))
    :ets.delete(tid)
  end

  bench "ets insert", [_unused: inspect_table(bench_context)] do
    tid = bench_context
    :ets.insert(tid, {:random.uniform(1000), :x})
    :ok
  end

  defp inspect_table(tid) do
    IO.inspect :ets.info(tid)
  end
end

Run time values

When you need to generate inputs for tests at run time without affecting the measurements and you can't use before_each_bench/1 hook for that, the following trick can be used:

# bench/string_bench.exs
defmodule StringBench do
  use Benchfella

  bench "reverse string", [str: gen_string()] do
    String.reverse(str)
  end

  defp gen_string() do
    String.duplicate("abc", 10000)
  end
end

mix bench.cmp

To compare results between multiple runs, use mix bench.cmp.

# Run 'mix bench' one more time.
# Each run automatically saves a snapshot in bench/snapshots.
$ mix bench
...

# 'mix bench.cmp' will read the two latest snapshots by default.
# You could also pass the snapshot files to compare as arguments.
$ mix bench.cmp -f percent
bench/snapshots/2015-03-26T01:17:17.snapshot vs
bench/snapshots/2015-03-26T01:19:30.snapshot

## ListBench
reverse list              -10.32%

## StringBench
reverse string dynamic    +2.26%
reverse string            +3.33%

mix bench.graph

Benchfella can produce an HTML page with graphs providing various insights into the raw data obtained from running mix bench.

# run the benchmarks twice
$ mix bench
...
$ mix bench
...

# 'mix bench.graph' works similarly to 'mix bench.cmp' except it can display
# all given snapshots on one graph.
$ mix bench.graph
Wrote bench/graphs/index.html

$ open bench/graphs/index.html

License

This software is licensed under the MIT license.