LLM-as-judge grounder using atomic claim decomposition.
Asks an LLM to decompose the answer into atomic factual claims and
verify each claim against the retrieved chunks. Each claim comes back
with a verdict (supported, unsupported, or contradicted) and the
1-based indices of the chunks that support it. Claims are then mapped
back to byte offsets in the answer for span-level highlighting.
This follows the RAGAS faithfulness pattern: one structured-output LLM
call instead of token-level NLI scoring. Compared to
Arcana.Grounder.Hallmark, this grounder:
- Does a single LLM call regardless of answer length or chunk count.
- Uses semantic reasoning instead of token-overlap NLI, which handles paraphrase and synthesis better.
- Returns chunk attribution directly from the LLM rather than via a secondary word-overlap pass.
Requirements
Requires req_llm. The judge model defaults to
Application.get_env(:arcana, :judge_model), falling back to
"anthropic:claude-haiku-4-5".
Usage
Pipeline.ground(ctx, grounder: Arcana.Grounder.LLMJudge)
Loop.run(question, grounder: Arcana.Grounder.LLMJudge)Options
:judge_model- Model spec passed to ReqLLM. Defaults toApplication.get_env(:arcana, :judge_model, "anthropic:claude-haiku-4-5").:judge_temperature- Sampling temperature. Defaults to0.0for deterministic verdicts.:judge_max_tokens- Max output tokens. Defaults to2048.:judge_fn- Override the LLM call entirely with a 3-arity function(question, answer, chunks) -> {:ok, %{claims: [...]}} | {:error, term}. Used for tests to avoid network calls.