Tribunal.TestCase (Tribunal v3.0.0)

Copy Markdown View Source

Represents a single evaluation test case.

Fields

  • input - The JSON-compatible input passed to the system under test (required)
  • evaluation_input - Optional text representation shown to judges
  • actual_output - The LLM response to evaluate (required for evaluation)
  • expected_output - Golden/ideal answer for comparison (optional)
  • context - Ground truth context for faithfulness checks (optional)
  • retrieval_context - Actual retrieved docs from RAG (optional)
  • metadata - Additional info like latency, tokens, cost (optional)

Example

test_case = %Tribunal.TestCase{
  input: "What's the return policy?",
  actual_output: "You can return items within 30 days.",
  context: ["Returns accepted within 30 days with receipt."],
  expected_output: "Items can be returned within 30 days with a receipt."
}

Summary

Types

json_scalar()
json_value()
t()

Functions

display_input(test_case)

Returns a safe, human-readable representation of an input for reports and test names.

evaluation_input(test_case)

Returns the text judges should evaluate for a test case.

new(attrs)

Creates a new test case from a map or keyword list.

validate(test_case)

Validates the fields required before a test case is passed to a provider or evaluator.

validate_input(value)

Validates that an input can be represented in JSON without changing its shape.

with_metadata(test_case, metadata)

Adds metadata (latency, tokens, cost, etc).

with_output(test_case, output)

Sets the actual output on an existing test case. Useful when the dataset provides input/context but output comes from your LLM.

with_retrieval_context(test_case, context)

Sets the retrieval context from your RAG pipeline.

Types

json_scalar()

@type json_scalar() :: String.t() | number() | boolean() | nil

json_value()

@type json_value() ::
  json_scalar() | [json_value()] | %{required(String.t()) => json_value()}

t()

@type t() :: %Tribunal.TestCase{
  actual_output: String.t() | nil,
  context: [String.t()] | String.t() | nil,
  evaluation_input: String.t() | nil,
  expected_output: String.t() | nil,
  input: json_value(),
  metadata: map() | nil,
  retrieval_context: [String.t()] | nil
}

Functions

display_input(test_case)

@spec display_input(t() | term()) :: String.t()

Returns a safe, human-readable representation of an input for reports and test names.

evaluation_input(test_case)

@spec evaluation_input(t()) :: String.t()

Returns the text judges should evaluate for a test case.

An explicit evaluation_input wins. String inputs remain unchanged and structured inputs use their JSON representation.

new(attrs)

Creates a new test case from a map or keyword list.

Examples

Tribunal.TestCase.new(input: "Hello", actual_output: "Hi there!")
Tribunal.TestCase.new(%{"input" => "Hello", "actual_output" => "Hi!"})

validate(test_case)

@spec validate(t()) :: :ok | {:error, String.t()}

Validates the fields required before a test case is passed to a provider or evaluator.

validate_input(value)

@spec validate_input(term()) :: :ok | {:error, String.t()}

Validates that an input can be represented in JSON without changing its shape.

with_metadata(test_case, metadata)

Adds metadata (latency, tokens, cost, etc).

with_output(test_case, output)

Sets the actual output on an existing test case. Useful when the dataset provides input/context but output comes from your LLM.

with_retrieval_context(test_case, context)

Sets the retrieval context from your RAG pipeline.