Statistics for the answer/reflect-judge eval lane.
The matched-pair eval used to report bare point estimates (e.g. "reflect 0.700, fixed 0.700, delta 0.000"), which is underpowered at n=30 (a binomial proportion there has a ±0.08 standard error) and treats two coincident point estimates as "equal means". These helpers replace that with:
wilson_interval/2— a 95% Wilson score interval per arm, so 0.700 reads as 0.700 [~0.52, ~0.83] and the width makes the uncertainty explicit. Wilson is used over Wald because it stays inside [0, 1] at the boundaries (0% / 100%).mcnemar_exact/2/paired/1— because both arms answer the SAME instances, the reflect-vs-fixed comparison is PAIRED. Only the discordant pairs carry signal; the exact binomial McNemar test gives an honest p-value where naive "delta of two independent proportions" would not.replicate_summary/1— mean, sample SD, and range across repeated runs, to characterize the A/A nondeterminism noise floor (the minimum delta worth believing) before attributing a difference to a code change.
Pure and side-effect-free.
Summary
Functions
Exact two-sided McNemar p-value for discordant paired counts b and c
(b = A-correct/B-wrong, c = A-wrong/B-correct). Uses the binomial exact test on
min(b, c) successes out of b + c trials at rate 0.5 — no chi-square
approximation, so it is valid at the small n this eval runs at. b + c == 0
(no discordant pairs) returns 1.0.
Summarize a list of {a_correct?, b_correct?} matched pairs (arm A = e.g. fixed,
arm B = e.g. reflect). Returns accuracies with Wilson CIs, the discordant counts,
the paired delta (B − A), and the exact McNemar p-value.
Mean, sample standard deviation, and range over replicate accuracies — the A/A
noise floor. A single value yields sd: 0.0 (no spread estimable).
95% Wilson score interval {lo, hi} for successes out of n.
Functions
@spec mcnemar_exact(non_neg_integer(), non_neg_integer()) :: float()
Exact two-sided McNemar p-value for discordant paired counts b and c
(b = A-correct/B-wrong, c = A-wrong/B-correct). Uses the binomial exact test on
min(b, c) successes out of b + c trials at rate 0.5 — no chi-square
approximation, so it is valid at the small n this eval runs at. b + c == 0
(no discordant pairs) returns 1.0.
Summarize a list of {a_correct?, b_correct?} matched pairs (arm A = e.g. fixed,
arm B = e.g. reflect). Returns accuracies with Wilson CIs, the discordant counts,
the paired delta (B − A), and the exact McNemar p-value.
Mean, sample standard deviation, and range over replicate accuracies — the A/A
noise floor. A single value yields sd: 0.0 (no spread estimable).
@spec wilson_interval(non_neg_integer(), non_neg_integer(), keyword()) :: {float(), float()}
95% Wilson score interval {lo, hi} for successes out of n.
n == 0 returns {0.0, 1.0} (no information). opts[:z] overrides the critical
value (default 1.96, i.e. 95%).