How judging works

What happens between Cite.judge/4 and the report, with the requests the client receives at each step.

flowchart LR
    A[Screen] --> B[Gather]
    B --> C[Judge]
    C --> D[Report]

Cite.judge/4 is the shell: it validates the options into a Cite.Run, calls the client, halves an oversized window, and assembles the Cite.Report. Every decision lives in the three pure modules, which never see the client.

Within a round, the shell sends up to concurrency requests at once, 4 by default. Round 2 starts only when round 1 has finished. Given the same replies, the report is the same at any setting: errors, usage, and models keep window and finding order.

Round 1: screen

Every detect — filters, directly screened concerns, factors — is asked of every passage, up to window passages per request. A request's state holds its window under the source's as, keyed by id. The passages stay in source order because the model reads neighbours. All the request's questions share that state. Each question's inspect names one passage's text:

{
  "state": {
    "turns": {
      "t14": {"id": "t14", "speaker": "customer", "text": "if this keeps happening I'm switching"},
      "t15": {"id": "t15", "speaker": "agent", "text": "Let me check the line from here."}
    }
  },
  "questions": {
    "t14:cancellation_risk": {
      "type": "noul",
      "instructions": {
        "question": "Does `turns.t14.text` say the customer is thinking of cancelling or switching provider?",
        "inspect": "`turns.t14.text`"
      },
      "criteria": {"true": {"what": "..."}, "false": {"what": "...", "not_for": "..."}}
    }
  }
}

Question keys are "<passage id>:<detect>". A score above threshold is a match. Matches decide what is gathered, and so what every confirm reads. That is why threshold changes verdicts, not only recall (see How a finding is read).

A request the provider refuses as :request_too_large is halved and both halves sent. Only a lone passage over the cap becomes an error; the rest of its window is still screened.

Between rounds: gather

Fixed rules, no model call:

• A passage that fails any filter is set aside. So is a passage whose window failed: it has no scores.
• A directly screened concern gathers every remaining match into one finding, however many. A passage that matches several concerns is evidence for each.
• A concern built from factors fills each role with its strongest match. A distinct role takes the strongest match no other role holds. A required role left empty means no finding.

Round 2: judge

One request per finding.

A directly screened concern puts its passages under as, so they give each other context. Confirms are keyed confirm:<passage id>; descriptors compare every passage:

{
  "state": {
    "turns": {
      "t14": {"id": "t14", "speaker": "customer", "text": "if this keeps happening I'm switching"},
      "t31": {"id": "t31", "speaker": "customer", "text": "honestly I've been looking at other providers"}
    }
  },
  "questions": {
    "confirm:t14": {"type": "noul", "instructions": {"question": "Does `turns.t14.text` evidence ...", "inspect": "`turns.t14.text`"}, "criteria": {"...": "..."}},
    "confirm:t31": {"type": "noul", "instructions": {"question": "Does `turns.t31.text` evidence ...", "inspect": "`turns.t31.text`"}, "criteria": {"...": "..."}},
    "severity": {"type": "score", "instructions": {"question": "...", "compare": ["`turns.t14.text`", "`turns.t31.text`"]}, "criteria": ["...", "...", "..."]}
  }
}

A concern built from factors puts each role at the top level and asks the checks that apply:

{
  "state": {
    "earlier_fix": {"id": "t3", "speaker": "customer", "text": "an engineer came out and fixed it in March"},
    "fault": {"id": "t9", "speaker": "customer", "text": "it's dropping every evening again"}
  },
  "questions": {
    "same_fault": {
      "type": "noul",
      "instructions": {
        "question": "Do `earlier_fix.text` and `fault.text` describe the same fault?",
        "compare": ["`earlier_fix.text`", "`fault.text`"]
      },
      "criteria": {"true": {"what": "..."}, "false": {"what": "..."}}
    },
    "severity": {"type": "score", "instructions": {"question": "...", "compare": ["`earlier_fix.text`", "`fault.text`"]}, "criteria": ["...", "...", "..."]}
  }
}

A round-2 request refused as too large becomes that finding's error. It is not split: each confirm reads its passage beside all the others, so a split would judge a different finding (see How a finding is read).

Verdict rules

With {low, high} as the review band:

• Evidence of a directly screened concern. A passage whose confirm is at or below low is dropped; below high it is cited as :review; at or above high, as :holds.
• Evidence of a concern built from factors. Every passage filling a role is cited as :holds, once, even if it fills several roles.
• :fails: any asked check at or below low, or no evidence left.
• :holds: every asked check at or above high, and at least one citation holds.
• :review: everything else.

How a finding is read

A confirm does not judge its passage alone. It reads the passage for its part in the whole finding, beside every other passage gathered with it. So a line that refers back to another ("it's doing it again") can hold beside the line it refers to and fail alone. This was measured in an internal benchmark of conversation transcripts, where lines lean on turns far away. A source of self-contained passages may behave differently. Three things follow:

• A finding's evidence is judged as a set. Show a finding's citations together. One citation shown alone is something the model never judged alone.
• Gathering shapes judgment. threshold decides which passages are gathered, and so what every confirm reads. A passage near the threshold can move another passage's verdict across the review band from run to run: round-1 noise reaches round 2.
• A finding is one request. Round 2 never splits a finding, so a larger finding is a larger request, not more of them.

Replies

The client's reply is checked once, in the shell. A reply that skips a question, or answers one with a value its question cannot have, is no verdict on that question. The whole request becomes a Cite.Error with {:missing_answers, keys} or {:malformed_answers, keys}, and nothing from it is read. A Noul or a confidence must be a probability from 0 to 1, a Score a number from 0 to its last level's index, between levels included, and a Choice one of its options.