Foresight.Evals.Stats (Foresight v0.1.0)

Copy Markdown View Source

Statistics for the answer/reflect-judge eval lane.

The matched-pair eval used to report bare point estimates (e.g. "reflect 0.700, fixed 0.700, delta 0.000"), which is underpowered at n=30 (a binomial proportion there has a ±0.08 standard error) and treats two coincident point estimates as "equal means". These helpers replace that with:

  • wilson_interval/2 — a 95% Wilson score interval per arm, so 0.700 reads as 0.700 [~0.52, ~0.83] and the width makes the uncertainty explicit. Wilson is used over Wald because it stays inside [0, 1] at the boundaries (0% / 100%).
  • mcnemar_exact/2 / paired/1 — because both arms answer the SAME instances, the reflect-vs-fixed comparison is PAIRED. Only the discordant pairs carry signal; the exact binomial McNemar test gives an honest p-value where naive "delta of two independent proportions" would not.
  • replicate_summary/1 — mean, sample SD, and range across repeated runs, to characterize the A/A nondeterminism noise floor (the minimum delta worth believing) before attributing a difference to a code change.

Pure and side-effect-free.

Summary

Functions

Exact two-sided McNemar p-value for discordant paired counts b and c (b = A-correct/B-wrong, c = A-wrong/B-correct). Uses the binomial exact test on min(b, c) successes out of b + c trials at rate 0.5 — no chi-square approximation, so it is valid at the small n this eval runs at. b + c == 0 (no discordant pairs) returns 1.0.

Summarize a list of {a_correct?, b_correct?} matched pairs (arm A = e.g. fixed, arm B = e.g. reflect). Returns accuracies with Wilson CIs, the discordant counts, the paired delta (B − A), and the exact McNemar p-value.

Mean, sample standard deviation, and range over replicate accuracies — the A/A noise floor. A single value yields sd: 0.0 (no spread estimable).

95% Wilson score interval {lo, hi} for successes out of n.

Functions

mcnemar_exact(b, c)

@spec mcnemar_exact(non_neg_integer(), non_neg_integer()) :: float()

Exact two-sided McNemar p-value for discordant paired counts b and c (b = A-correct/B-wrong, c = A-wrong/B-correct). Uses the binomial exact test on min(b, c) successes out of b + c trials at rate 0.5 — no chi-square approximation, so it is valid at the small n this eval runs at. b + c == 0 (no discordant pairs) returns 1.0.

paired(pairs)

@spec paired([{boolean(), boolean()}]) :: map()

Summarize a list of {a_correct?, b_correct?} matched pairs (arm A = e.g. fixed, arm B = e.g. reflect). Returns accuracies with Wilson CIs, the discordant counts, the paired delta (B − A), and the exact McNemar p-value.

replicate_summary(values)

@spec replicate_summary([number()]) :: map()

Mean, sample standard deviation, and range over replicate accuracies — the A/A noise floor. A single value yields sd: 0.0 (no spread estimable).

wilson_interval(successes, n, opts \\ [])

@spec wilson_interval(non_neg_integer(), non_neg_integer(), keyword()) ::
  {float(), float()}

95% Wilson score interval {lo, hi} for successes out of n.

n == 0 returns {0.0, 1.0} (no information). opts[:z] overrides the critical value (default 1.96, i.e. 95%).