Benchee Hex VersiondocsInline docsBuild StatusCoverage Status

Library for easy and nice (micro) benchmarking in Elixir. It allows you to compare the performance of different pieces of code at a glance. Benchee is also versatile and extensible, relying only on functions - no macros! There are also a bunch of plugins to draw pretty graphs and more!

Benchee runs each of your functions for a given amount of time after an initial warmup. It uses the raw run times it could gather in that time to show different statistical values like average, iterations per second and the standard deviation.

Benchee has a nice and concise main interface, and its behavior can be altered through lots of configuration options:

list = Enum.to_list(1..10_000)

map_fun = fn(i) -> [i, i * i] end



Benchee.run(%{

  "flat_map"    => fn -> Enum.flat_map(list, map_fun) end,

  "map.flatten" => fn -> list |> Enum.map(map_fun) |> List.flatten end

}, time: 10)

Produces the following output on the console:

tobi@speedy ~/github/benchee $ mix run samples/run.exs

Elixir 1.4.0

Erlang 19.1

Benchmark suite executing with the following configuration:

warmup: 2.0s

time: 10.0s

parallel: 1

inputs: none specified

Estimated total run time: 24.0s



Benchmarking flat_map...

Benchmarking map.flatten...



Name                  ips        average  deviation         median

flat_map           2.29 K      437.22 μs    ±17.32%      418.00 μs

map.flatten        1.28 K      778.50 μs    ±15.92%      767.00 μs



Comparison:

flat_map           2.29 K

map.flatten        1.28 K - 1.78x slower

The aforementioned plugins like benchee_html make it possible to generate nice looking html reports, where individual graphs can also be exported as PNG images:

report

Features

Provides you with the following statistical data:

Benchee does not:

Benchee only has a small runtime dependency on deep_merge for merging configuration and is aimed at being the core benchmarking logic. Further functionality is provided through plugins that then pull in dependencies, such as HTML generation and CSV export. Check out the available plugins!

Installation

Add benchee to your list of dependencies in mix.exs:

defp deps do

  [{:benchee, "~> 0.6", only: :dev}]

end

Install via mix deps.get and then happy benchmarking as described in Usage :)

Usage

After installing just write a little Elixir benchmarking script:

list = Enum.to_list(1..10_000)

map_fun = fn(i) -> [i, i * i] end



Benchee.run(%{

  "flat_map"    => fn -> Enum.flat_map(list, map_fun) end,

  "map.flatten" => fn -> list |> Enum.map(map_fun) |> List.flatten end

})

This produces the following output:

tobi@speedy ~/github/benchee $ mix run samples/run.exs

Elixir 1.4.0

Erlang 19.1

Benchmark suite executing with the following configuration:

warmup: 2.0s

time: 5.0s

parallel: 1

inputs: none specified

Estimated total run time: 14.0s



Benchmarking flat_map...

Benchmarking map.flatten...



Name                  ips        average  deviation         median

flat_map           2.28 K      438.07 μs    ±16.66%      419.00 μs

map.flatten        1.25 K      802.99 μs    ±13.40%      782.00 μs



Comparison:

flat_map           2.28 K

map.flatten        1.25 K - 1.83x slower

See Features for a description of the different statistical values and what they mean.

If you're looking to see how to make something specific work, please refer to the samples directory. Also, especially when wanting to extend benchee check out the hexdocs.

Configuration

Benchee takes a wealth of configuration options, in the most common Benchee.run/2 interface these are passed as the second argument in the form of an optional keyword list:

Benchee.run(%{"some function" => fn -> magic end}, print: [benchmarking: false])

The available options are the following (also documented in hexdocs).

Inputs

:inputs is a very useful configuration that allows you to run the same benchmarking with different inputs. Functions can have different performance characteristics on differently shaped inputs be that structure or input size.

One of such cases is comparing tail-recursive and body-recursive implementations of map. More information in the repository with the benchmark and the blog post.

map_fun = fn(i) -> i + 1 end

inputs = %{

  "Small (1 Thousand)"    => Enum.to_list(1..1_000),

  "Middle (100 Thousand)" => Enum.to_list(1..100_000),

  "Big (10 Million)"      => Enum.to_list(1..10_000_000),

}



Benchee.run %{

  "map tail-recursive" =>

    fn(list) -> MyMap.map_tco(list, map_fun) end,

  "stdlib map" =>

    fn(list) -> Enum.map(list, map_fun) end,

  "map simple body-recursive" =>

    fn(list) -> MyMap.map_body(list, map_fun) end,

  "map tail-recursive different argument order" =>

    fn(list) -> MyMap.map_tco_arg_order(list, map_fun) end

}, time: 15, warmup: 5, inputs: inputs

Omitting some of the output this produces the following results:

##### With input Big (10 Million) #####

Name                                                  ips        average  deviation         median

map tail-recursive different argument order          5.09      196.48 ms     ±9.70%      191.18 ms

map tail-recursive                                   3.86      258.84 ms    ±22.05%      246.03 ms

stdlib map                                           2.87      348.36 ms     ±9.02%      345.21 ms

map simple body-recursive                            2.85      350.80 ms     ±9.03%      349.33 ms



Comparison:

map tail-recursive different argument order          5.09

map tail-recursive                                   3.86 - 1.32x slower

stdlib map                                           2.87 - 1.77x slower

map simple body-recursive                            2.85 - 1.79x slower



##### With input Middle (100 Thousand) #####

Name                                                  ips        average  deviation         median

stdlib map                                         584.79        1.71 ms    ±16.20%        1.67 ms

map simple body-recursive                          581.89        1.72 ms    ±15.38%        1.68 ms

map tail-recursive different argument order        531.09        1.88 ms    ±17.41%        1.95 ms

map tail-recursive                                 471.64        2.12 ms    ±18.93%        2.13 ms



Comparison:

stdlib map                                         584.79

map simple body-recursive                          581.89 - 1.00x slower

map tail-recursive different argument order        531.09 - 1.10x slower

map tail-recursive                                 471.64 - 1.24x slower



##### With input Small (1 Thousand) #####

Name                                                  ips        average  deviation         median

stdlib map                                        66.10 K       15.13 μs    ±58.17%       15.00 μs

map tail-recursive different argument order       62.46 K       16.01 μs    ±31.43%       15.00 μs

map simple body-recursive                         62.35 K       16.04 μs    ±60.37%       15.00 μs

map tail-recursive                                55.68 K       17.96 μs    ±30.32%       17.00 μs



Comparison:

stdlib map                                        66.10 K

map tail-recursive different argument order       62.46 K - 1.06x slower

map simple body-recursive                         62.35 K - 1.06x slower

map tail-recursive                                55.68 K - 1.19x slower

As you can see, the tail-recursive approach is significantly faster for the Big 10 Million input while body recursion outperforms it or performs just as well on the Middle and Small inputs.

Therefore, I highly recommend using this feature and checking different realistically structured and sized inputs for the functions you benchmark!

Formatters

Among all the configuration options, one that you probably want to use are the formatters. Formatters are functions that take one argument (the benchmarking suite with all its results) and then generate some output. You can specify multiple formatters to run for the benchmarking run.

So if you are using the HTML plugin and you want to run both the console formatter and the HTML formatter this looks like this (after you installed it of course):

list = Enum.to_list(1..10_000)

map_fun = fn(i) -> [i, i * i] end



Benchee.run(%{

  "flat_map"    => fn -> Enum.flat_map(list, map_fun) end,

  "map.flatten" => fn -> list |> Enum.map(map_fun) |> List.flatten end

},

  formatters: [

    &Benchee.Formatters.HTML.output/1,

    &Benchee.Formatters.Console.output/1

  ],

  html: [file: "samples_output/my.html"],

)

Setup and teardown

If you want to do setup and teardown, i.e. do something before or after the benchmarking suite executes this is very easy in benchee. As benchee is just plain old functions just do it before/after you call benchee:

your_setup()



Benchee.run %{"Awesome stuff" => fn -> magic end }



your_teardown()

More verbose usage

It is important to note that the benchmarking code shown before is the convenience interface. The same benchmark in its more verbose form looks like this:

list = Enum.to_list(1..10_000)

map_fun = fn(i) -> [i, i * i] end



Benchee.init(time: 3)

|> Benchee.system

|> Benchee.benchmark("flat_map", fn -> Enum.flat_map(list, map_fun) end)

|> Benchee.benchmark("map.flatten",

                     fn -> list |> Enum.map(map_fun) |> List.flatten end)

|> Benchee.measure

|> Benchee.statistics

|> Benchee.Formatters.Console.output

This is a take on the functional transformation of data applied to benchmarks here:

  1. Configure the benchmarking suite to be run
  2. Define the functions to be benchmarked
  3. Run n benchmarks with the given configuration gathering raw run times per function
  4. Generate statistics based on the raw run times
  5. Format the statistics in a suitable way
  6. Output the formatted statistics

This is also part of the official API and allows for more fine grained control. (It's also what Benchee does internally when you use Benchee.run/2).

Do you just want to have all the raw run times? Just work with the result of Benchee.measure/1! Just want to have the calculated statistics and use your own formatting? Grab the result of Benchee.statistics/1! Or, maybe you want to write to a file or send an HTTP post to some online service? Just use the Benchee.Formatters.Console.format/1 and then send the result where you want.

This way Benchee should be flexible enough to suit your needs and be extended at will. Have a look at the available plugins.

Plugins

Packages that work with Benchee to provide additional functionality.

With the HTML plugin for instance you can get fancy graphs like this boxplot:

boxplot

Of course there also are normal bar charts including standard deviation:

flat_map_ips

Presentation + general benchmarking advice

If you're into watching videos of conference talks and also want to learn more about benchmarking in general I can recommend watching my talk from ElixirLive 2016. Slides can be found here, video - click the washed out image below ;)

Benchee Video

Contributing

Contributions to Benchee are very welcome! Bug reports, documentation, spelling corrections, whole features, feature ideas, bugfixes, new plugins, fancy graphics... all of those (and probably more) are much appreciated contributions!

Please respect the Code of Conduct.

You can get started with a look at the open issues.

A couple of (hopefully) helpful points:

Development