Foresight.Evals.Labeling.Judge (Foresight v0.1.0)

Copy Markdown View Source

D2 component 4: LLM graded-relevance judge. For each (query, candidate) pair it asks a complete_json_schema/4-capable LLM (default: z.ai GLM-5.2) for a 0..3 relevance grade, and emits source: :llm label attrs recording the judge model.

Integrity contract: unlike the entity extractor, there is NO safe fallback grade — a fabricated default would bias D2's optimization. So on an incapable/absent LLM, any error, an unparseable result, or a grade outside 0..3, the pair is SKIPPED and reported under :errors (with its query_ref/memory_ref and a reason), never labeled. The bulk pass therefore reports exactly what it graded and what it could not.

The pair's split is preserved on the emitted label so a :calibration pair's llm row lines up with its human row for the Cohen's-kappa join. Structure + parsing are pure and fake-LLM testable; only the live call is paid.

Summary

Functions

The rubric new runs grade against by default.

Every rubric version this judge can still grade against.

Types

pair()

@type pair() :: %{
  :bank_id => String.t(),
  :query_ref => String.t(),
  :memory_ref => String.t(),
  optional(atom()) => term()
}

result()

@type result() :: %{labels: [map()], errors: [map()]}

Functions

label_all(pairs, opts)

@spec label_all(
  [pair()],
  keyword()
) :: result()

rubric_version()

The rubric new runs grade against by default.

rubric_versions()

Every rubric version this judge can still grade against.