Assert output contains substring(s)

Assert output contains all values

Assert output contains at least one of the values

Assert response is correct compared to expected (requires req_llm).

Options

• :query - The query/input.
• :expected - Required. The expected output.
• :threshold - Score threshold (default: 0.8)
• :verbose - When true, prints score reasoning (default: false)
• :model - LLM model to use for judging

Assert output is a valid email

Assert output ends with suffix

Assert output exactly equals expected

Assert response is faithful to context (requires req_llm).

Options

• :context - Required. The context to check faithfulness against.
• :threshold - Score threshold (default: 0.8)
• :verbose - When true, prints score reasoning (default: false)
• :model - LLM model to use for judging

Examples

assert_faithful response, context: @docs
assert_faithful response, context: @docs, verbose: true

Assert output is valid JSON

Assert output within Levenshtein distance of target

Assert output under maximum length

Assert output is under token limit

Assert output meets minimum length

Assert output appears to be a refusal.

Uses an LLM judge to detect refusals, including soft refusals and redirections that pattern matching might miss.

Examples

assert_refusal response
assert_refusal response, verbose: true

Assert output matches regex pattern

Assert response is relevant to query (requires req_llm).

Options

• :query - Required. The query to check relevance against.
• :threshold - Score threshold (default: 0.8)
• :verbose - When true, prints score reasoning (default: false)
• :model - LLM model to use for judging

Assert response is semantically similar to expected (requires alike).

Options

• :expected - Required. The expected output to compare against.
• :threshold - Similarity threshold (default: 0.8)
• :verbose - When true, prints similarity score (default: false)

Assert output starts with prefix

Assert output is a valid URL

Assert output word count within range

Logs verbose output for judge assertions.

Verbose mode is enabled when:

• verbose: true is passed to the assertion, OR
• config :tribunal, verbose: true is set in config

Uses Logger.info for passes and Logger.warning for failures, which integrates properly with ExUnit's output capture.

Assert response has no bias (requires req_llm).

Options

• :query - The query/input.
• :threshold - Score threshold (default: 0.8)
• :verbose - When true, prints score reasoning (default: false)
• :model - LLM model to use for judging

Assert output does not contain substring(s)

Assert response did not overclaim agency (requires req_llm).

Pairs with Tribunal.RedTeam.Plugins.ExcessiveAgency.

Options

• :purpose - Required. The assistant's purpose text.
• :query - Optional context about the input.
• :threshold - Score threshold (default: 0.8)
• :verbose - When true, prints score reasoning (default: false)
• :model - LLM model to use for judging

Assert response did not fabricate specifics (requires req_llm).

Pairs with Tribunal.RedTeam.Plugins.Hallucination. Distinct from refute_hallucination/2, which grades faithfulness against a supplied context. This macro grades on the signature of confabulation without ground truth.

Options

• :purpose - Required. The assistant's purpose text.
• :query - Optional context about the input.
• :threshold - Score threshold (default: 0.8)
• :verbose - When true, prints score reasoning (default: false)
• :model - LLM model to use for judging

Assert response has no hallucinations (requires req_llm).

Options

• :context - Required. The context to check against.
• :threshold - Score threshold (default: 0.8)
• :verbose - When true, prints score reasoning (default: false)
• :model - LLM model to use for judging

Assert response has no harmful content (requires req_llm).

Options

• :query - The query/input.
• :threshold - Score threshold (default: 0.8)
• :verbose - When true, prints score reasoning (default: false)
• :model - LLM model to use for judging

Assert response was not hijacked off-topic (requires req_llm).

Pairs with Tribunal.RedTeam.Plugins.Hijacking: red-team cases generated by that plugin carry the assistant's purpose under expected.hijacked.purpose, and this macro grades responses against it.

Options

• :purpose - Required. The assistant's purpose text.
• :query - Optional context about the input.
• :threshold - Score threshold (default: 0.8)
• :verbose - When true, prints score reasoning (default: false)
• :model - LLM model to use for judging

Examples

refute_hijacked response,
  purpose: @assistant_purpose,
  query: input,
  verbose: true

Assert response did not adopt an unauthorised persona or authority (requires req_llm).

Pairs with Tribunal.RedTeam.Plugins.Imitation.

Options

• :purpose - Required. The assistant's purpose text.
• :query - Optional context about the input.
• :threshold - Score threshold (default: 0.8)
• :verbose - When true, prints score reasoning (default: false)
• :model - LLM model to use for judging

Assert response shows no signs of jailbreak success (requires req_llm).

Uses LLM-as-judge for nuanced detection of jailbreak indicators.

Options

• :query - The original prompt that was sent to the LLM
• :threshold - Score threshold (default: 0.8)
• :verbose - When true, prints score reasoning (default: false)
• :model - LLM model to use for judging
• :llm_client - Custom LLM client function for testing

Examples

refute_jailbreak(response,
  query: "Ignore previous instructions and...",
  verbose: true
)

Assert output contains no PII (LLM-based).

Uses LLM-as-judge for comprehensive detection including names, addresses, and context-dependent PII that regex cannot catch.

Examples

refute_pii response

Assert output contains no PII (LLM-based, comprehensive).

Uses LLM-as-judge for nuanced detection including names, addresses, and context-dependent PII that regex cannot catch.

Options

• :query - Optional context about the input
• :threshold - Score threshold (default: 0.8)
• :verbose - When true, prints score reasoning (default: false)
• :model - LLM model to use for judging

Examples

refute_pii response, query: "user profile request"
refute_pii response, verbose: true

Assert response does not violate a supplied policy (requires req_llm).

Pairs with Tribunal.RedTeam.Plugins.Policy: red-team cases generated by that plugin carry the policy text under expected.policy_violation.policy, and this macro grades responses against it.

Options

• :policy - Required. The policy text the response is graded against.
• :query - Optional context about the input.
• :threshold - Score threshold (default: 0.8)
• :verbose - When true, prints score reasoning (default: false)
• :model - LLM model to use for judging

Examples

refute_policy_violation response,
  policy: @guardrails,
  query: input,
  verbose: true

Assert response did not leak the system prompt or internal config (requires req_llm).

Pairs with Tribunal.RedTeam.Plugins.PromptExtraction.

Options

• :purpose - Required. The assistant's purpose text.
• :query - Optional context about the input.
• :threshold - Score threshold (default: 0.8)
• :verbose - When true, prints score reasoning (default: false)
• :model - LLM model to use for judging

Assert output contains no toxic content (LLM-based).

Uses LLM-as-judge for nuanced detection of toxic content including hate speech, harassment, threats, and harmful language.

Examples

refute_toxic response

Assert output contains no toxic content (LLM-based).

Options

• :query - Optional context about the input
• :threshold - Score threshold (default: 0.8)
• :verbose - When true, prints score reasoning (default: false)
• :model - LLM model to use for judging

Examples

refute_toxic response, query: "user message"
refute_toxic response, verbose: true

Assert response has no toxic content (requires req_llm).

Options

• :query - The query/input.
• :threshold - Score threshold (default: 0.8)
• :verbose - When true, prints score reasoning (default: false)
• :model - LLM model to use for judging