Foresight.Evals.Dataset (Foresight v0.1.0)

Copy Markdown View Source

The dataset-adapter seam for the honest evaluation harness.

Every eval dataset — LongMemEval now, a license-clean BEIR slice later — is normalised to a list of Instances so the runner and the ranking metrics are dataset-agnostic. An adapter's only job is to turn its native format into Instances; nothing downstream knows or cares which corpus it came from.

An Instance carries everything the runner needs and nothing it doesn't:

  • memories — retain-ready maps (each with a stable "id", mirrored into metadata["eval_id"] so recall results can be mapped back to their label).
  • relevance — graded gain per memory id (0 irrelevant … n ideal). Scoring reads ids and labels only, never answer text, so a corpus phrase earns nothing.
  • reference_answer / abstention? — for grading reflect (a correct answer, or a correct refusal when the evidence is absent).

Summary

Types

t()

Functions

label_memories(memories)

Stamp each memory's stable "id" into metadata["eval_id"] so recall results (which echo metadata) can be mapped back to their relevance label. Idempotent.

pool_distractors(instances, k, seed \\ 1)

Inject k distractor memories into each instance, sampled deterministically from the memories of other instances (relevance 0). This turns an evidence-only split (where every candidate is relevant, so retrieval is trivial) into a distractor-heavy retrieval task — the setting the full LongMemEval haystack provides natively. Distractor ids are namespaced so they never collide with, or accidentally become, a relevant id.

ranked_eval_ids(results)

Extract the ranked list of eval ids from a recall response's results.

Types

t()

@type t() :: %Foresight.Evals.Dataset{
  abstention?: boolean(),
  id: String.t(),
  memories: [map()],
  query: String.t(),
  query_timestamp: String.t() | nil,
  question_type: String.t() | nil,
  reference_answer: String.t() | nil,
  relevance: %{optional(String.t()) => number()}
}

Functions

label_memories(memories)

@spec label_memories([map()]) :: [map()]

Stamp each memory's stable "id" into metadata["eval_id"] so recall results (which echo metadata) can be mapped back to their relevance label. Idempotent.

pool_distractors(instances, k, seed \\ 1)

@spec pool_distractors([t()], pos_integer(), integer()) :: [t()]

Inject k distractor memories into each instance, sampled deterministically from the memories of other instances (relevance 0). This turns an evidence-only split (where every candidate is relevant, so retrieval is trivial) into a distractor-heavy retrieval task — the setting the full LongMemEval haystack provides natively. Distractor ids are namespaced so they never collide with, or accidentally become, a relevant id.

Deterministic in seed: no wall-clock or global RNG, so the eval is reproducible.

ranked_eval_ids(results)

@spec ranked_eval_ids([map()]) :: [String.t()]

Extract the ranked list of eval ids from a recall response's results.