Loads evaluation datasets from JSON or YAML files.
Dataset Format
Each item in the dataset should have:
input- The query/prompt (required)context- Ground truth context (optional)expected_output- Golden answer (optional)expected- Assertions to run (optional)
Example JSON
[
{
"input": "What's the return policy?",
"context": "Returns accepted within 30 days.",
"expected": {
"contains": ["30 days"],
"faithful": {"threshold": 0.8}
}
}
]Example YAML
- input: What's the return policy?
context: Returns accepted within 30 days.
expected:
contains:
- 30 days
faithful:
threshold: 0.8
Summary
Functions
Loads a dataset from a file path.
Loads a dataset, raising on error.
Loads a dataset and extracts assertions per test case.
Loads with assertions, raising on error.
Functions
Loads a dataset from a file path.
Returns {:ok, [test_cases]} or {:error, reason}.
Loads a dataset, raising on error.
Loads a dataset and extracts assertions per test case.
Returns {:ok, [{test_case, assertions}]}.
Loads with assertions, raising on error.