Benchee Hex VersiondocsInline docsBuild Status

Library for easy and nice (micro) benchmarking in Elixir. It allows you to compare the performance of different pieces of code and functions at a glance. Benchee is also versatile and extensible, relying only on functions - no macros!

Somewhat inspired by benchmark-ips from the ruby world, but a very different interface and a functional spin.

General features:

Provides you with the following statistical data:

Benchee does not:

Benchee has no runtime dependencies and is aimed at being the core benchmarking logic. Further functionality is provided through plugins that then pull in dependencies, such as CSV export. Check out the available plugins!

Installation

When available in Hex, the package can be installed as:

Add benchee to your list of dependencies in mix.exs:

def deps do
  [{:benchee, "~> 0.4", only: :dev}]
end

Install via mix deps.get and then happy benchmarking as described in Usage :)

Usage

After installing just write a little Elixir benchmarking script:

list = Enum.to_list(1..10_000)
map_fun = fn(i) -> [i, i * i] end

Benchee.run(%{time: 3}, %{
  "flat_map"    => fn -> Enum.flat_map(list, map_fun) end,
  "map.flatten" => fn -> list |> Enum.map(map_fun) |> List.flatten end})

First configuration options are passed, the only options available so far are:

Running this script produces an output like:

tobi@happy ~/github/benchee $ mix run samples/run.exs
Erlang/OTP 19 [erts-8.0] [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false]
Elixir 1.3.2
Benchmark suite executing with the following configuration:
warmup: 2.0s
time: 3.0s
parallel: 1
Estimated total run time: 10.0s

Benchmarking flat_map...
Benchmarking map.flatten...

Name                  ips        average    deviation         median
map.flatten        989.80      1010.31μs    (±12.63%)       998.00μs
flat_map           647.35      1544.75μs    (±10.54%)      1556.00μs

Comparison:
map.flatten        989.80
flat_map           647.35 - 1.53x slower

See the general description for the meaning of the different statistics.

It is important to note that the benchmarking code shown before is the convenience interface. The same benchmark in its more verbose form looks like this:

list = Enum.to_list(1..10_000)
map_fun = fn(i) -> [i, i * i] end

Benchee.init(%{time: 3})
|> Benchee.benchmark("flat_map", fn -> Enum.flat_map(list, map_fun) end)
|> Benchee.benchmark("map.flatten",
                     fn -> list |> Enum.map(map_fun) |> List.flatten end)
|> Benchee.measure
|> Benchee.statistics
|> Benchee.Formatters.Console.output

This is a take on the functional transformation of data applied to benchmarks here:

  1. Configure the benchmarking suite to be run
  2. run n benchmarks with the given configuration gathering raw run times per function (done in 2 steps, gathering the benchmarks and then running them with Benchee.measure)
  3. Generate statistics based on the raw run times
  4. Format the statistics in a suitable way
  5. Output the formatted statistics

This is also part of the official API and allows for more fine grained control. Do you just want to have all the raw run times? Grab them before Benchee.statistics! Just want to have the calculated statistics and use your own formatting? Grab the result of Benchee.statistics! Or, maybe you want to write to a file or send an HTTP post to some online service? Just replace the IO.puts.

This way Benchee should be flexible enough to suit your needs and be extended at will. Have a look at the available plugins.

For more example usages and benchmarks have a look at the samples directory!

Formatters

Among all the configuration options, one that you probably want to use are the formatters. Formatters are functions that take one argument (the benchmarking suite with all its results) and then generate some output. You can specify multiple formatters to run for the benchmarking run.

So if you are using the CSV plugin and you want to run both the console formatter and the CSV formatter this looks like this:

list = Enum.to_list(1..10_000)
map_fun = fn(i) -> [i, i * i] end

Benchee.run(
  %{
    formatters: [
      &Benchee.Formatters.CSV.output/1,
      &Benchee.Formatters.Console.output/1
    ],
    csv: %{file: "my.csv"}
  },
  %{
    "flat_map"    => fn -> Enum.flat_map(list, map_fun) end,
    "map.flatten" => fn -> list |> Enum.map(map_fun) |> List.flatten end
  })

Plugins

Packages that work with Benchee to provide additional functionality.

(You didn't really expect to find tons of plugins here when the library was just released, did you? ;) )

Contributing

Contributions to Benchee are very welcome! Bug reports, documentation, spelling corrections, whole features, feature ideas, bugfixes, new plugins, fancy graphics... all of those (and probably more) are much appreciated contributions!

You can get started with a look at the open issues.

A couple of (hopefully) helpful points:

Development