Foresight.Evals.Labeling.GoldSet (Foresight v0.1.0)

Copy Markdown View Source

D2 component 3a: build the human-graded gold set from per-query candidate pools and serialize it to a gradeable artifact.

build/2 selects a stratified sample of (query, candidate) pairs (balanced across question types), deterministically under a seed, caps how many pairs any one query contributes, and assigns each pair a :calibration split (co-graded by the LLM judge → Cohen's kappa) or a :test split (held out from D2 training, so the final exceed metric rests on human labels). to_artifact/2 renders the selection to a JSON-encodable map whose "grade" fields are blank for the human to fill.

Determinism mirrors the LongMemEval stratified sampler: :rand.seed(:exsss, {s, s*7, s*13}). Pure: no DB, no recall call.

Summary

Types

gold_pair()

Functions

build(query_pools, opts \\ [])

Build the gold set from query_pools (each %{query_ref, query_text, question_type, candidates: [pooled candidate]}).

to_artifact(gold_pairs, bank_id)

Render a built gold set to a JSON-encodable artifact with blank grades.

Types

gold_pair()

@type gold_pair() :: %{
  query_ref: String.t(),
  query_text: String.t(),
  question_type: String.t() | nil,
  memory_ref: String.t(),
  content: String.t() | nil,
  context: String.t() | nil,
  split: :calibration | :test
}

Functions

build(query_pools, opts \\ [])

@spec build(
  [map()],
  keyword()
) :: [gold_pair()]

Build the gold set from query_pools (each %{query_ref, query_text, question_type, candidates: [pooled candidate]}).

Options: :seed, :target_pairs, :per_query_cap, :calibration_ratio.

to_artifact(gold_pairs, bank_id)

@spec to_artifact([gold_pair()], String.t()) :: map()

Render a built gold set to a JSON-encodable artifact with blank grades.