Evaluation

This library makes measured claims, so it owes you the method. This guide is mostly about how to disbelieve the numbers efficiently.

The headline, with its uncertainty

Matched-reader benchmark on LongMemEval-oracle. Both arms use the same reader model, the same judge, and the same 30 instances; the only variable is the memory architecture. Three replicates:

McNemar exact p = 0.227 / 1.000 / 0.774 — non-significant in all three.

Verdict: parity. The mean delta sits inside the ±0.1 band that n=30 nondeterminism produces, and the paired test cannot separate the arms in any replicate.

The most useful number in that table is the delta's range: +0.034 to +0.166, across identical configuration. Replicate 1 alone reads like a clear win. Publishing it would have been wrong. This is why the project's own rule is ≥3 replicates before any claim, and why a single run of this benchmark should never persuade you of anything.

Two metrics, two different questions

Answer accuracy — did reflect answer correctly? The number that matters, and the one with a ceiling problem: a consolidation change that produced +52% more observations moved it 0.711 → 0.708. The lane cannot see improvements it should plausibly be able to see, so a null result here is weak evidence of no effect.

nDCG@10 / MRR — did recall put the right memories at the top? Far more sensitive, because it grades ranking directly instead of waiting for a downstream model to notice. This is how the fusion default was chosen (+0.1009 nDCG@10, CI excluding zero).

They answer different questions and do not substitute for one another. The fusion change is a demonstrated ranking win that did not yield a significant accuracy lead. Both statements are true and this documentation keeps them apart.

Where the labels come from

Relevance labels are agent-anchored, not human-graded. Stated plainly because an eval number is worth exactly what its labels are worth.

• Agent-produced labels carry source: :agent — a distinct provenance from :human, so nothing downstream can mistake one for the other.
• LLM-judged labels carry source: :llm and stamp judge_model as "<model>@<rubric-version>", derived from the rubric actually sent. There is deliberately no caller override for that field: one existed, and it let a label claim a rubric version the judge never saw.
• Agreement is gated on Cohen's quadratic-weighted kappa ≥ 0.6.

The known risk this leaves open: a model-judged relevance label could share a bias with the ranking change being evaluated. That threatens the magnitude of the +0.1009, not its direction, and the honest remedy is a human-labelled subset rather than a code change. If you need human-anchored ground truth for a publication or regulatory claim, this repository does not have it, and the kappa gate does not manufacture it.

The harness refuses to score a broken run

retain degrades rather than failing when the LLM times out: extraction falls back to storing raw content, consolidation skips a scope after exhausting retries. Correct for production, disastrous for a benchmark — the arm silently gets worse while everything reports healthy.

This is not hypothetical. One head-to-head ran with 10 extraction fallbacks and 5 skipped merges — the arm was storing raw blobs instead of extracted facts, which is precisely the deficit worth −0.267 accuracy — and the harness would have printed a confident scoreboard had it not independently run out of wall-clock.

Foresight.Evals.DegradationWatch now counts both via telemetry, and the benchmark asserts zero after printing its report: the numbers stay visible so a failed run is diagnosable, but the test fails so they cannot be quoted as a measurement.

Reproducing

FORESIGHT_LME_ANSWER_MODEL=... \
FORESIGHT_LME_JUDGE_MODEL=... \
FORESIGHT_LME_PER_TYPE=5 \
FORESIGHT_LME_SEED=42 \
mix test test/foresight/evals/longmemeval_reflect_judge_eval_test.exs --include ml

Set FORESIGHT_LME_HINDSIGHT_BASE_URL to add the Hindsight arm. Full method, every historical result including the unflattering ones, and the reasoning behind each, live in docs/foresight_bench_gc2541.md.

Two practical notes, learned expensively. Reader choice dominates: a 32B model on modest hardware took 15–22 hours per replicate where a hosted model took 18 minutes and cost $0.84. And both arms must run the same reader — a mismatched reader measures the models, not the memory.

What has not been measured

Latency, throughput, memory footprint, and stability under load — absolutely or comparatively. Every number in this project is answer quality. Since accuracy is now at parity, those are the dimensions where the two implementations might actually differ, and nobody has looked yet.