Pure, deterministic retrieval-quality metrics — the gaming-resistant core of the Track B eval.
These operate on IDs and graded relevance labels only. They never inspect answer text, so they cannot be satisfied by emitting a corpus phrase (the failure mode that let the old phrase-matching grader be gamed from inside the ranking code). A system scores well here only by ordering the genuinely-relevant memories ahead of the irrelevant ones.
Relevance labels are non-negative graded gains (e.g. 0 = irrelevant, 1 = related,
2 = relevant, 3 = ideal). ranked_ids is the system's output order (most-relevant
first); relevance maps an id to its graded label (absent ⇒ 0).
Summary
Functions
Aggregate a metric across many queries by simple mean. Each element of per_query is a
float; returns 0.0 for an empty list.
MRR — reciprocal rank of the first result whose relevance meets min_relevance
(default 1). 0.0 if no such result appears in ranked_ids.
nDCG@k — normalized discounted cumulative gain over the top k results.
Precision@k — fraction of the top k results that are relevant (≥ min_relevance).
Denominator is min(k, length(ranked_ids)) so a short result list is not penalised for
positions it never had.
Recall@k — fraction of the relevant items (relevance ≥ min_relevance) that appear in
the top k. 0.0 when nothing is relevant (nothing to recall).
Types
Functions
Aggregate a metric across many queries by simple mean. Each element of per_query is a
float; returns 0.0 for an empty list.
MRR — reciprocal rank of the first result whose relevance meets min_relevance
(default 1). 0.0 if no such result appears in ranked_ids.
@spec ndcg_at_k([id()], relevance(), pos_integer()) :: float()
nDCG@k — normalized discounted cumulative gain over the top k results.
DCG uses the standard log2(rank+1) discount. Normalized by the ideal DCG (the same labels sorted by descending gain), so 1.0 means "perfect ordering of the top k" and the score is comparable across queries with different label distributions. Returns 0.0 when there is no attainable gain (no relevant items), which is the correct floor.
@spec precision_at_k([id()], relevance(), pos_integer(), number()) :: float()
Precision@k — fraction of the top k results that are relevant (≥ min_relevance).
Denominator is min(k, length(ranked_ids)) so a short result list is not penalised for
positions it never had.
@spec recall_at_k([id()], relevance(), pos_integer(), number()) :: float()
Recall@k — fraction of the relevant items (relevance ≥ min_relevance) that appear in
the top k. 0.0 when nothing is relevant (nothing to recall).