Statistical comparison functions for benchmark results.
Pure math — no external dependencies.
Summary
Functions
Fisher exact test p-value for 2x2 contingency table (two-tailed).
Sample size per group for comparing two proportions.
Wilson score confidence interval for a proportion.
Functions
@spec fisher_exact_p( non_neg_integer(), non_neg_integer(), non_neg_integer(), non_neg_integer() ) :: float()
Fisher exact test p-value for 2x2 contingency table (two-tailed).
Compares pass/fail counts between two variants. Uses log-factorials to avoid overflow.
Examples
iex> p = PtcRunner.Metrics.Statistics.fisher_exact_p(5, 5, 5, 5)
iex> p == 1.0
true@spec sample_size_for_two_proportions(float(), float(), float(), float()) :: pos_integer()
Sample size per group for comparing two proportions.
Uses the standard formula for desired power and significance level. Returns the number of observations needed per group.
Examples
iex> n = PtcRunner.Metrics.Statistics.sample_size_for_two_proportions(0.5, 0.6)
iex> is_integer(n) and n > 0
true@spec wilson_interval(non_neg_integer(), pos_integer(), float()) :: {float(), float()}
Wilson score confidence interval for a proportion.
Returns {lower, upper} bounds. Uses z=1.96 for 95% confidence (default).
Examples
iex> {lower, upper} = PtcRunner.Metrics.Statistics.wilson_interval(7, 10)
iex> is_float(lower) and is_float(upper)
true