Ragex.
View Source
Retrieval quality evaluator for comparing search strategies.
Computes standard IR metrics — NDCG, MRR, Precision@K, Recall@K — against a golden query set, making it possible to measure whether a retrieval change improves or degrades quality.
Metrics
- NDCG@K — Normalised Discounted Cumulative Gain: rewards relevant results that appear earlier in the ranked list. Scores are graded (0–3) when multiple relevance levels are defined, or binary (0/1) otherwise.
- MRR — Mean Reciprocal Rank: mean of 1/rank for the first relevant result across all queries. Good proxy for "do users find something useful quickly?"
- Precision@K — fraction of the top-K results that are relevant.
- Recall@K — fraction of all known-relevant results that appear in top-K.
Golden query set format
[
%{
query: "function that retries HTTP requests",
relevant: [
%{node_type: :function, node_id: {MyModule, :retry, 2}, grade: 3},
%{node_type: :function, node_id: {MyModule, :with_retry, 1}, grade: 2}
]
},
...
]grade is optional (default 1). Grades are used for NDCG when present.
A/B comparison
strategy_a = fn query -> Hybrid.search(query, strategy: :fusion, limit: 10) end
strategy_b = fn query -> Hybrid.search(query, strategy: :fusion, rerank: true, limit: 10) end
Evaluator.compare(golden_queries, strategy_a, strategy_b, k: 10)Usage
golden = Evaluator.load_golden("priv/eval/golden_queries.json")
results = Evaluator.run(golden, fn q -> Hybrid.search(q, limit: 10) end, k: 10)
IO.inspect(results.ndcg) # => 0.72Summary
Functions
Run two search functions against the same golden set and return a diff report.
Load a golden query set from a JSON file.
Compute MRR for a single query (reciprocal rank of first relevant result).
Compute NDCG@K for a single query.
Compute Precision@K for a single query.
Compute Recall@K for a single query.
Run a search function against a golden query set and return aggregate metrics.
Types
@type golden_query() :: %{query: String.t(), relevant: [relevant_item()]}
@type grade() :: 0 | 1 | 2 | 3
@type metrics() :: %{ ndcg: float(), mrr: float(), precision_at_k: float(), recall_at_k: float(), query_count: non_neg_integer(), k: pos_integer() }
@type search_fn() :: (String.t() -> {:ok, [result_item()]} | {:error, term()})
Functions
@spec compare([golden_query()], search_fn(), search_fn(), keyword()) :: map()
Run two search functions against the same golden set and return a diff report.
Returns a map with keys :strategy_a, :strategy_b, and :delta (B minus A).
Positive delta values mean strategy B improved.
@spec load_golden(String.t()) :: {:ok, [golden_query()]} | {:error, term()}
Load a golden query set from a JSON file.
Expected format: a JSON array of {"query": "...", "relevant": [...]} objects.
Relevant items may have "grade" (integer 0–3, default 1) alongside
"node_type" (string) and "node_id" (any JSON value).
@spec mrr(results :: [result_item()], relevant :: [relevant_item()]) :: float()
Compute MRR for a single query (reciprocal rank of first relevant result).
@spec ndcg( results :: [result_item()], relevant :: [relevant_item()], k :: pos_integer() ) :: float()
Compute NDCG@K for a single query.
results is a ranked list of %{node_type, node_id} maps.
relevant is the golden set with optional grade values.
@spec precision_at_k( results :: [result_item()], relevant :: [relevant_item()], k :: pos_integer() ) :: float()
Compute Precision@K for a single query.
@spec recall_at_k( results :: [result_item()], relevant :: [relevant_item()], k :: pos_integer() ) :: float()
Compute Recall@K for a single query.
@spec run([golden_query()], search_fn(), keyword()) :: metrics()
Run a search function against a golden query set and return aggregate metrics.
Options
:k— cutoff rank (default: 10):verbose— log per-query scores (default: false)