Tribunal.Dataset (Tribunal v1.4.0)

Copy Markdown View Source

Loads evaluation datasets from JSON or YAML files.

Dataset Format

Each item in the dataset should have:

  • input - The query/prompt (required)
  • context - Ground truth context (optional)
  • expected_output - Golden answer (optional)
  • expected - Assertions to run (optional)

Example JSON

[
  {
    "input": "What's the return policy?",
    "context": "Returns accepted within 30 days.",
    "expected": {
      "contains": ["30 days"],
      "faithful": {"threshold": 0.8}
    }
  }
]

Example YAML

- input: What's the return policy?
  context: Returns accepted within 30 days.
  expected:
    contains:
      - 30 days
    faithful:
      threshold: 0.8

Summary

Functions

Loads a dataset from a file path.

Loads a dataset, raising on error.

Loads a dataset and extracts assertions per test case.

Loads with assertions, raising on error.

Functions

load(path)

Loads a dataset from a file path.

Returns {:ok, [test_cases]} or {:error, reason}.

load!(path)

Loads a dataset, raising on error.

load_with_assertions(path)

Loads a dataset and extracts assertions per test case.

Returns {:ok, [{test_case, assertions}]}.

load_with_assertions!(path)

Loads with assertions, raising on error.