Pure, robust statistics for benchmark mode (mix excessibility.debug --benchmark=N).

Benchmark mode loops a test N times and collects, per run, a map of measurement key to duration_ms. summarize/2 turns those per-run samples into a value-free benchmark.json artifact.

Timing contract

Timing is diagnostic, never comparable across environments and never part of the digest. A key being free of outliers means "no relative outlier in these samples," not "fast." Robust stats (median + MAD) are used deliberately instead of mean/stddev so a single slow run cannot skew the picture.

Sample 1 is treated as cold (first-run costs: compilation, connection warmup, cold caches). Samples 2..N are warm. Cold and warm are reported separately per the timing contract; mixing them would poison the median.

Input shape

samples is a list (one entry per run, in run order) of maps:

[
  %{"MyLive/mount" => 12.0, "query:sha256:aaa" => 5.0},
  %{"MyLive/mount" => 6.0,  "query:sha256:aaa" => 3.0},
  ...
]

Keys are opaque strings (the mix task uses "<view>/<callback>" and "query:<fingerprint>"); values are duration_ms numbers. Keys may vary between runs — each key is summarized over the runs in which it appears.

Output shape

%{
  schema: "excessibility.benchmark/v1",
  runs: n,
  cold: %{key => %{median: number, mad: number, sample: number}},  # run 1 only
  warm: %{key => %{median: number, mad: number, samples: integer}}, # runs 2..n
  outliers: [
    %{key: String.t(), run: pos_integer(), value: number,
      median: number, mad: number, threshold: number}
  ],
  notes: [String.t()]
}

outliers is advisory only — a warm sample must clear three gates to be flagged, so scheduler/timer jitter in the sub-millisecond band is not reported as actionable evidence:

1. the robust statistical threshold value > median + k * mad (k defaults to 6, :k);
2. a minimum absolute effect value - median >= min_abs_ms (defaults to 1.0 ms, :min_abs_ms); and
3. a minimum relative effect value >= median * min_rel_factor (defaults to 1.5, :min_rel_factor).

Set min_abs_ms: 0.0, min_rel_factor: 1.0 to restore pure-statistical flagging. Each outlier carries weak_evidence: true when its key has fewer than 5 warm samples, and a run-level notes entry labels the whole artifact as weak evidence when there are too few warm runs for stable MAD inference. It never encodes a pass/fail verdict; the raw value, median, mad, threshold, and 1-based run index are attached so a reader can judge for themselves.