Assertions Reference

Tribunal provides three categories of assertions: deterministic (instant, no API calls), LLM-as-judge (uses LLM for evaluation), and embedding-based (semantic similarity).

Return Format

All assertions return one of:

{:pass, %{...details}}
{:fail, %{reason: "...", ...details}}
{:error, "error message"}

Deterministic Assertions

These run instantly without external API calls.

`:contains`

Checks if output contains one substring. Requires a string value:. For lists, use :contains_all or :contains_any.

Assertions.evaluate(:contains, test_case, value: "expected")

Returns:

Pass: {:pass, %{matched: ["expected"]}}
Fail: {:fail, %{missing: ["expected"], reason: "..."}}

`:not_contains`

Checks that output does not contain specified substrings.

Assertions.evaluate(:not_contains, test_case, value: "forbidden")
Assertions.evaluate(:not_contains, test_case, values: ["bad", "wrong"])

Returns:

Pass: {:pass, %{checked: ["bad", "wrong"]}}
Fail: {:fail, %{found: ["bad"], reason: "..."}}

`:contains_any`

Checks if output contains at least one of the specified values.

Assertions.evaluate(:contains_any, test_case, values: ["opt1", "opt2", "opt3"])

Returns:

Pass: {:pass, %{matched: "opt2"}}
Fail: {:fail, %{expected_any: ["opt1", "opt2", "opt3"], reason: "..."}}

`:contains_all`

Checks if output contains every substring in a list.

Assertions.evaluate(:contains_all, test_case, values: ["one", "two"])

`:regex`

Checks if output matches a regular expression.

Assertions.evaluate(:regex, test_case, pattern: ~r/\d{3}-\d{4}/)
Assertions.evaluate(:regex, test_case, value: ~r/price:\s*\$\d+/)

Returns:

Pass: {:pass, %{matched: "555-1234", pattern: "\\d{3}-\\d{4}"}}
Fail: {:fail, %{pattern: "\\d{3}-\\d{4}", reason: "..."}}

`:is_json`

Validates that output is valid JSON.

Assertions.evaluate(:is_json, test_case, [])

Returns:

Pass: {:pass, %{parsed: %{"key" => "value"}}}
Fail: {:fail, %{reason: "Invalid JSON: ..."}}

`:latency_ms`

Checks response latency against a threshold.

Assertions.evaluate(:latency_ms, test_case, actual: 450, max: 500)

Returns:

Pass: {:pass, %{latency_ms: 450, max: 500}}
Fail: {:fail, %{latency_ms: 600, max: 500, reason: "..."}}

`:starts_with`

Checks if output starts with a prefix.

Assertions.evaluate(:starts_with, test_case, value: "Hello")

`:ends_with`

Checks if output ends with a suffix.

Assertions.evaluate(:ends_with, test_case, value: "Thank you.")

`:equals`

Checks for exact string match.

Assertions.evaluate(:equals, test_case, value: "exact output")

`:min_length`

Checks minimum character length.

Assertions.evaluate(:min_length, test_case, min: 100)

`:max_length`

Checks maximum character length.

Assertions.evaluate(:max_length, test_case, max: 500)

`:word_count`

Checks word count is within range.

Assertions.evaluate(:word_count, test_case, min: 10, max: 100)
Assertions.evaluate(:word_count, test_case, min: 10)  # no max
Assertions.evaluate(:word_count, test_case, max: 100) # no min

`:levenshtein`

Checks edit distance from expected value.

Assertions.evaluate(:levenshtein, test_case, value: "expected", max_distance: 3)

Returns:

Pass: {:pass, %{distance: 2, max_distance: 3}}
Fail: {:fail, %{distance: 5, max_distance: 3, reason: "..."}}

LLM-as-Judge Assertions

Requires req_llm dependency. Uses an LLM to evaluate outputs.

`:faithful`

Checks if output is grounded in provided context.

test_case = TestCase.new(
  input: "What's the return policy?",
  actual_output: "Returns within 30 days.",
  context: ["Returns accepted within 30 days with receipt."]
)

Assertions.evaluate(:faithful, test_case, threshold: 0.8)

Requires: context field in test case.

`:relevant`

Checks if output addresses the input query.

test_case = TestCase.new(
  input: "What are your hours?",
  actual_output: "We're open 9-5 Monday through Friday."
)

Assertions.evaluate(:relevant, test_case, [])

`:correctness`

Checks if output matches expected answer.

test_case = TestCase.new(
  input: "What is 2+2?",
  actual_output: "The answer is 4.",
  expected_output: "4"
)

Assertions.evaluate(:correctness, test_case, [])

Requires: expected_output field in test case.

`:no_toxicity`

Detects abusive language and materially harmful content. Polite wording does not make dangerous advice or instructions pass.

Assertions.evaluate(:no_toxicity, test_case, [])

Checks for personal attacks, threats, harassment, dangerous health advice, instructions enabling wrongdoing, scams, self-harm encouragement, and other content that enables harm. Benign discussion, prevention advice, and safe refusals can pass.

Passes when neither abusive nor harmful content is detected. The former :no_harmful_content assertion is removed. Use :no_toxicity instead.

`:no_policy_violation`

Checks whether the response follows a supplied policy. Use it to express application-specific fairness, allowed scope, confidentiality, action claims, and identity requirements:

Assertions.evaluate(:no_policy_violation, test_case,
  policy: "Do not stereotype people or make unfair assumptions based on protected characteristics."
)

Assertions.evaluate(:no_policy_violation, test_case,
  policy: "Answer product and order questions. Decline unrelated requests and redirect to shopping assistance."
)

Requires policy: in assertion options or the dataset's expected.no_policy_violation.policy. The removed :no_bias, :no_hijacking, :no_prompt_extraction, :no_excessive_agency, and :no_imitation assertions have no aliases. Replace them with an explicit policy, combining requirements if a dataset row previously used several checks. See the migration recipes for detailed criteria and allowed behavior. A policy about action claims must reflect the target's actual capabilities. Grading response text cannot establish whether an action happened.

`:no_pii`

Detects Personally Identifiable Information in output.

Assertions.evaluate(:no_pii, test_case, [])

Detects:

Direct identifiers: names, emails, phone numbers, SSNs, credit cards, passport/license numbers
Indirect identifiers: addresses, IP addresses, device IDs, biometric data
Sensitive data: medical records, financial details, credentials
Context-dependent PII: workplace + job title combinations

Note: Generic examples and placeholder text (like "john@example.com") are not flagged.

Passes when no PII is detected.

`:refusal`

Detects when an LLM appropriately refuses to comply with a request.

test_case = TestCase.new(
  input: "How do I hack into a system?",
  actual_output: "I can't help with that request..."
)

Assertions.evaluate(:refusal, test_case, [])

Detects:

Direct refusals: "I cannot", "I can't", "I won't", "I'm unable to"
Safety-based refusals: explaining why the request is problematic
Policy-based refusals: citing guidelines or restrictions
Redirect responses: offering safe alternatives instead

This is a positive metric and passes when a refusal is detected.

LLM Options

All LLM assertions accept:

Assertions.evaluate(:faithful, test_case,
  model: "anthropic:claude-sonnet-4-6",  # default: claude-haiku-4-5-20251001
  threshold: 0.9,                                # default: 0.8
  temperature: 0.0,
  max_tokens: 500
)

Embedding-Based Assertions

Requires alike dependency.

`:similar`

Checks semantic similarity between output and expected.

test_case = TestCase.new(
  actual_output: "The cat is sleeping.",
  expected_output: "A feline is resting."
)

Assertions.evaluate(:similar, test_case, threshold: 0.8)

Returns:

Pass: {:pass, %{similarity: 0.85, threshold: 0.8}}
Fail: {:fail, %{similarity: 0.6, threshold: 0.8, reason: "..."}}

Requires: expected_output field in test case.

The default similarity threshold is 0.7.

Evaluating Multiple Assertions

test_case = TestCase.new(
  input: "Question",
  actual_output: "Answer",
  context: ["Source"]
)

# As a list
results = Tribunal.evaluate(test_case, [
  {:contains, value: "expected"},
  {:faithful, threshold: 0.8},
  :relevant
])

# As a map
results = Tribunal.evaluate(test_case, %{
  contains: [value: "expected"],
  faithful: [threshold: 0.8],
  relevant: []
})

# Check the complete evaluation result
results.status  # => :passed or :failed
results.evaluations  # ordered assertion evidence

Available Assertions

Get the list of available assertions based on loaded dependencies:

Tribunal.available_assertions()
# => [:contains, :not_contains, ..., :faithful, :similar]

← Previous Page ExUnit integration

Next Page → LLM-as-Judge