ExAbby.Statistics (ex_abby v0.5.1)

Copy Markdown

Statistical comparisons for experiment summaries.

Conversion outcomes are binary per trial, so comparisons use the unique conversion count rather than the total number of recorded success events.

Summary

Functions

Compares several success metrics with control as one Holm correction family.

Compares each treatment in an experiment summary with the named control.

Functions

compare_metrics_to_control(summary, metrics, opts \\ [])

Compares several success metrics with control as one Holm correction family.

A treatment can be significant for any requested metric; the other metrics do not also need to be significant. Combining the metrics into one family keeps the overall false-positive rate at :alpha when any metric can identify a winner.

The returned map contains the same significance result as compare_to_control/3 for each requested metric. Every configured treatment and requested metric remains in the family even when it has insufficient data.

compare_to_control(summary, metric, opts \\ [])

Compares each treatment in an experiment summary with the named control.

The returned p-values invert two-sided beta-binomial mixture confidence sequences for Bernoulli outcomes. They remain valid under continuous monitoring and are adjusted together with Holm's method. The correction family is every treatment in summary, including those returned as :insufficient_data, so that the family is fixed by the experiment's design rather than by which arms currently have data.

Each comparison also includes an anytime-valid confidence sequence for treatment-minus-control lift. That interval is built at :alpha and is not Holm-adjusted, so with several treatments an interval can exclude zero while the same comparison's adjusted :p_value exceeds :alpha. Use :p_value for the decision and :confidence_interval for the magnitude.

A comparison is marked :insufficient_data only when either arm has no valid trials. Sparse conversion counts produce a wide confidence sequence instead of suppressing the result.

Options:

  • :control_name - control variation name, defaults to "control"
  • :alpha - significance threshold, defaults to 0.05