ExUnit-style assertion macros for LLM evaluation.
Summary
Functions
Assert response is correct compared to expected (requires req_llm).
Assert response is faithful to context (requires req_llm).
Assert output within Levenshtein distance of target
Assert output appears to be a refusal.
Assert response is relevant to query (requires req_llm).
Assert response is semantically similar to expected (requires alike).
Logs verbose output for passing assertions.
Assert output contains no PII (LLM-based).
Assert output contains no PII (LLM-based, comprehensive).
Assert response does not violate a supplied policy (requires req_llm).
Assert response has no abusive or harmful content (requires req_llm).
Functions
Assert response is correct compared to expected (requires req_llm).
Options
:query- The query/input.:expected- Required. The expected output.:threshold- Score threshold (default: 0.8):verbose- When true, logs score reasoning for passes (default: false):model- LLM model to use for judging
Assert response is faithful to context (requires req_llm).
Options
:context- Required. The context to check faithfulness against.:threshold- Score threshold (default: 0.8):verbose- When true, logs score reasoning for passes (default: false):model- LLM model to use for judging
Examples
assert_faithful response, context: @docs
assert_faithful response, context: @docs, verbose: trueAssert output within Levenshtein distance of target
Assert output appears to be a refusal.
Uses an LLM judge to detect refusals, including soft refusals and redirections that pattern matching might miss.
Examples
assert_refusal response
assert_refusal response, verbose: trueAssert response is relevant to query (requires req_llm).
Options
:query- Required. The query to check relevance against.:threshold- Score threshold (default: 0.8):verbose- When true, logs score reasoning for passes (default: false):model- LLM model to use for judging
Assert response is semantically similar to expected (requires alike).
Options
:expected- Required. The expected output to compare against.:threshold- Similarity threshold (default: 0.7):verbose- When true, logs similarity score for passes (default: false)
Logs verbose output for passing assertions.
Verbose mode is enabled when:
verbose: trueis passed to the assertion, ORconfig :tribunal, verbose: trueis set in config
Uses Logger.info for passes. Failures are left to ExUnit's assertion output so the reason appears only once.
Assert output contains no PII (LLM-based).
Uses LLM-as-judge for comprehensive detection including names, addresses, and context-dependent PII that regex cannot catch.
Examples
refute_pii responseAssert output contains no PII (LLM-based, comprehensive).
Uses LLM-as-judge for nuanced detection including names, addresses, and context-dependent PII that regex cannot catch.
Options
:query- Optional context about the input:threshold- Score threshold (default: 0.8):verbose- When true, logs score reasoning for passes (default: false):model- LLM model to use for judging
Examples
refute_pii response, query: "user profile request"
refute_pii response, verbose: trueAssert response does not violate a supplied policy (requires req_llm).
Pairs with Tribunal.RedTeam.Plugins.Policy: red-team cases generated by
that plugin carry the policy text under expected.no_policy_violation.policy,
and this macro grades responses against it.
Options
:policy- Required. The policy text the response is graded against.:query- Optional context about the input.:threshold- Score threshold (default: 0.8):verbose- When true, logs score reasoning for passes (default: false):model- LLM model to use for judging
Examples
refute_policy_violation response,
policy: @guardrails,
query: input,
verbose: trueAssert response has no abusive or harmful content (requires req_llm).
Includes dangerous advice and instructions even when phrased politely.
Application-specific fairness and scope rules belong in
refute_policy_violation/2 with an explicit policy.
Options
:query- The query/input.:threshold- Score threshold (default: 0.8):verbose- When true, logs score reasoning for passes (default: false):model- LLM model to use for judging