ExAbby. Statistics
(ex_abby v0.5.1)
Copy Markdown
Statistical comparisons for experiment summaries.
Conversion outcomes are binary per trial, so comparisons use the unique conversion count rather than the total number of recorded success events.
Summary
Functions
Compares several success metrics with control as one Holm correction family.
Compares each treatment in an experiment summary with the named control.
Functions
Compares several success metrics with control as one Holm correction family.
A treatment can be significant for any requested metric; the other metrics do
not also need to be significant. Combining the metrics into one family keeps
the overall false-positive rate at :alpha when any metric can identify a
winner.
The returned map contains the same significance result as
compare_to_control/3 for each requested metric. Every configured treatment
and requested metric remains in the family even when it has insufficient data.
Compares each treatment in an experiment summary with the named control.
The returned p-values invert two-sided beta-binomial mixture confidence
sequences for Bernoulli outcomes. They remain valid under continuous
monitoring and are adjusted together with Holm's method. The correction family
is every treatment in summary, including those returned as
:insufficient_data, so that the family is fixed by the experiment's design
rather than by which arms currently have data.
Each comparison also includes an anytime-valid confidence sequence for
treatment-minus-control lift. That interval is built at :alpha and is not
Holm-adjusted, so with several treatments an interval can exclude zero while
the same comparison's adjusted :p_value exceeds :alpha. Use :p_value for
the decision and :confidence_interval for the magnitude.
A comparison is marked :insufficient_data only when either arm has no valid
trials. Sparse conversion counts produce a wide confidence sequence instead
of suppressing the result.
Options:
:control_name- control variation name, defaults to"control":alpha- significance threshold, defaults to0.05