Evaluates answer quality using LLM-as-judge.
Two flavors are supported:
- Faithfulness — does the answer only contain claims supported by the retrieved context? Catches hallucinations. Does not compare to any ground-truth answer.
- Correctness — does the answer convey the same facts as a
reference (ground-truth) answer? Catches wrong answers that happen
to be faithful to the retrieved chunks. Requires a
reference_answeron the test case.
Use faithfulness when you have retrieved chunks but no ground truth. Use correctness when you have a ground-truth reference. Both are cheap LLM-as-judge calls; running them together is common.
Summary
Functions
Returns the default correctness prompt template.
Returns the default faithfulness prompt template.
Scores a candidate answer against a reference answer via LLM-as-judge.
Evaluates the faithfulness of an answer to the retrieved chunks.
Functions
Returns the default correctness prompt template.
Returns the default faithfulness prompt template.
Scores a candidate answer against a reference answer via LLM-as-judge.
This is a correctness check, not a faithfulness check. Use this when your test cases have ground-truth reference answers and you want to know "did the agent produce a correct answer" rather than just "did the agent stay grounded in whatever it retrieved."
Options
:llm- LLM function (required):prompt- Custom prompt functionfn question, answer, reference -> prompt end
Returns
{:ok, %{score: integer, reasoning: string | nil}}on success{:error, reason}on failure
Evaluates the faithfulness of an answer to the retrieved chunks.
Options
:llm- LLM function (required):prompt- Custom prompt functionfn question, chunks, answer -> prompt end
Returns
{:ok, %{score: integer, reasoning: string | nil}}on success{:error, reason}on failure