Tribunal.Dataset (Tribunal v3.0.0)

Copy Markdown View Source

Loads evaluation datasets from JSON or YAML files.

Dataset Format

Each item in the dataset should have:

  • input - The JSON-compatible input passed to the system under test (required)
  • evaluation_input - Optional text representation shown to judges
  • context - Ground truth context (optional)
  • expected_output - Golden answer (optional)
  • expected - Assertions to run (optional)

Example JSON

[
  {
    "input": "What's the return policy?",
    "context": "Returns accepted within 30 days.",
    "expected": {
      "contains": "30 days",
      "faithful": {"threshold": 0.8}
    }
  }
]

Example YAML

- input: What's the return policy?
  context: Returns accepted within 30 days.
  expected:
    contains: 30 days
    faithful:
      threshold: 0.8

Summary

Functions

Loads a dataset from a file path.

Loads a dataset, raising on error.

Loads a dataset and extracts assertions per test case.

Loads with assertions, raising on error.

Functions

load(path)

Loads a dataset from a file path.

Returns {:ok, [test_cases]} or {:error, reason}.

load!(path)

Loads a dataset, raising on error.

load_with_assertions(path)

Loads a dataset and extracts assertions per test case.

Returns {:ok, [{test_case, assertions}]}.

load_with_assertions!(path)

Loads with assertions, raising on error.