Benchee v0.2.0 Benchee.Statistics

Statistics related functionality that is meant to take the raw benchmark run times and then compute statistics like the average and the standard devaition.

Summary

Functions

Calculates statistical data based on a series of run times for a job in microseconds

Sorts the given jobs fastest to slowest by average

Takes a job suite with job run times, returns a map representing the statistics of the job suite as follows

Functions

job_statistics(run_times)

Calculates statistical data based on a series of run times for a job in microseconds.

Examples

iex> run_times = [200, 400, 400, 400, 500, 500, 700, 900]
iex> Benchee.Statistics.job_statistics(run_times)
%{average:       500.0,
  ips:           2000.0,
  std_dev:       200.0,
  std_dev_ratio: 0.4,
  std_dev_ips:   800.0,
  median:        450.0}
sort(jobs)

Sorts the given jobs fastest to slowest by average.

Examples

iex> second = {"Second", %{average: 200.0}}
iex> third  = {"Third",  %{average: 400.0}}
iex> first  = {"First",  %{average: 100.0}}
iex> Benchee.Statistics.sort([second, third, first])
[{"First",  %{average: 100.0}},
 {"Second", %{average: 200.0}},
 {"Third",  %{average: 400.0}}]
statistics(map)

Takes a job suite with job run times, returns a map representing the statistics of the job suite as follows:

  • average - average run time of the job in μs (the lower the better)
  • ips - iterations per second, how often can the given function be executed within one second (the higher the better)
  • std_dev - standard deviation, a measurement how much results vary (the higher the more the results vary)
  • std_dev_ratio - standard deviation expressed as how much it is relative to the average
  • std_dev_ips - the absolute standard deviation of iterations per second (= ips * std_dev_ratio)
  • median - when all measured times are sorted, this is the middle value (or average of the two middle values when the number of times is even). More stable than the average and somewhat more likely to be a typical you see.

Parameters

  • suite - the job suite represented as a map after running the measurements, required to have the run_times available under the run_times key

Examples

iex> run_times = [200, 400, 400, 400, 500, 500, 700, 900]
iex> suite = %{run_times: [{"My Job", run_times}]}
iex> Benchee.Statistics.statistics(suite)
[{"My Job",
  %{average:       500.0,
    ips:           2000.0,
    std_dev:       200.0,
    std_dev_ratio: 0.4,
    std_dev_ips:   800.0,
    median:        450.0}}]