Foresight.Evals.Labeling.CandidatePool (Foresight v0.1.0)

Copy Markdown View Source

D2 component 2: pool the pre-fusion recall arms into the deduped candidate set to be labeled for a query.

Input is the strategy_items map that escapes ranked_recall_items/6arm_name => ranked item list, each item shaped by memory_unit_to_recall_response/2 (string "id", "content", "context", and metadata["eval_id"] as the stable label key). We keep the top-K per arm, union across arms, dedup on the stable memory_ref (eval_id), and record EVERY arm's rank for each candidate (arm_ranks), plus the best rank and arm-name list.

arm_ranks is what makes the pool re-fusable offline: RRF scores a candidate from each arm's rank, so D2's coordinate-ascent can rescore a whole pool under a trial weighting without re-running recall. Pools recorded before this existed carry only best_rank and must be regenerated before they can be ascended against.

Candidates lacking a stable eval_id cannot be mapped to a reusable label, so they are dropped from the pool and returned under :dropped — the caller logs that as coverage loss rather than silently labeling against ephemeral ids. Pure: no DB, no recall call.

Summary

Functions

Pool strategy_items into %{candidates: [candidate], dropped: [raw_item]}.

Types

candidate()

@type candidate() :: %{
  memory_ref: String.t(),
  memory_id: String.t(),
  content: String.t() | nil,
  context: String.t() | nil,
  arms: [String.t()],
  arm_ranks: %{required(String.t()) => pos_integer()},
  arm_scores: %{required(String.t()) => float()},
  best_rank: pos_integer()
}

Functions

pool(strategy_items, opts \\ [])

@spec pool(
  %{optional(String.t()) => [map()]},
  keyword()
) :: %{candidates: [candidate()], dropped: [map()]}

Pool strategy_items into %{candidates: [candidate], dropped: [raw_item]}.

Options: :top_k (per-arm cap, default 20).