Foresight.Evals.Runner (Foresight v0.1.0)

Copy Markdown View Source

Dataset-agnostic eval runner. Given normalised Dataset.Instances and a recall_fun, it scores each query with the gaming-resistant ranking metrics and aggregates overall and per question_type.

recall_fun is injected so the same orchestration runs against the real Ecto+pgvector+embedder pipeline (the honest measurement) or a deterministic stub (to unit test the aggregation without a database). It receives a Dataset.Instance and returns the recall response's results list (ranked, each result echoing metadata["eval_id"]).

Summary

Types

recall_fun()

Functions

aggregate(rows)

Aggregate a list of per-query metric rows into means.

format_report(map, opts \\ [])

Render a report table (overall + per question_type) as a string.

run(instances, recall_fun, opts \\ [])

Run the harness. Options: :k (nDCG/precision depth, default 10), :recall_k (default 5). Returns %{overall: metrics, by_type: %{type => metrics}, per_query: [...]} where metrics is %{n, ndcg, mrr, recall}.

Types

recall_fun()

@type recall_fun() :: (Foresight.Evals.Dataset.t() -> [map()])

Functions

aggregate(rows)

@spec aggregate([map()]) :: map()

Aggregate a list of per-query metric rows into means.

format_report(map, opts \\ [])

@spec format_report(
  map(),
  keyword()
) :: String.t()

Render a report table (overall + per question_type) as a string.

run(instances, recall_fun, opts \\ [])

@spec run([Foresight.Evals.Dataset.t()], recall_fun(), keyword()) :: map()

Run the harness. Options: :k (nDCG/precision depth, default 10), :recall_k (default 5). Returns %{overall: metrics, by_type: %{type => metrics}, per_query: [...]} where metrics is %{n, ndcg, mrr, recall}.