Cohen's kappa between human and llm calibration grades — the trust gate on the bulk judge. If the judge doesn't agree well enough with the human gold slice, its labels can't be trusted to train fusion weights against (optimizing to a machine's biases is the same metric-gaming trap the ranking metrics were built to avoid), so the pipeline gates the full labeling run on this number.

Grades are the ordinal 0..3 relevance scale, so :quadratic weighting (the default) penalizes an off-by-3 disagreement 9x more than off-by-1 — the right model for graded relevance. :linear and :unweighted are available for comparison. Pure math.