File-Based Evaluation Suites

Versioned suite manifests make a persisted Aludel evaluation reproducible from a repository, script, or CI job. A manifest selects the suite, prompt version, provider, and optional repeated-sampling policy. The database remains authoritative for test cases, document associations, dataset provenance, and quality policies.

YAML manifest

Create evals/support-answer.yaml:

schema_version: 1
suite_id: 9a756a58-eaec-43ca-99e6-f5c016d85d0c
prompt_version_id: e74cf2e1-94b6-4bcb-9ed9-b259661be906
provider_id: e1c60ec0-6d55-419b-b958-7d088055254f
sampling:
  samples: 5
  reducer: majority

Replace the example UUIDs with IDs from your Aludel installation, then run:

mix aludel.eval --file evals/support-answer.yaml

.yaml and .yml extensions are supported.

JSON manifest

JSON uses the same schema:

{
  "schema_version": 1,
  "suite_id": "9a756a58-eaec-43ca-99e6-f5c016d85d0c",
  "prompt_version_id": "e74cf2e1-94b6-4bcb-9ed9-b259661be906",
  "provider_id": "e1c60ec0-6d55-419b-b958-7d088055254f",
  "sampling": {
    "samples": 5,
    "reducer": "minimum_pass_rate",
    "minimum_pass_rate": 0.8
  }
}

Run it the same way:

mix aludel.eval --file evals/support-answer.json

Sampling options

sampling is optional. Without it, each test case runs once and must pass.

samples accepts integers from 1 through 20. The limit prevents an accidental manifest change from creating unbounded provider traffic.

For example, require four of five attempts to pass:

sampling:
  samples: 5
  reducer: minimum_pass_rate
  minimum_pass_rate: 0.8

Reports and quality gates

Reporter flags stay on the command line, so the same manifest can serve local output and different CI systems:

mix aludel.eval \
  --file evals/support-answer.yaml \
  --format junit \
  --output aludel-junit.xml

mix aludel.eval \
  --file evals/support-answer.yaml \
  --format github

The task exits unsuccessfully when the manifest or target records are invalid, execution fails, the suite is empty, or the effective quality decision does not pass. The suite's latest immutable quality policy is snapshotted at execution time. Without a policy, every persisted test case must pass.

--file is mutually exclusive with --suite-id, --prompt-version-id, and --provider-id. Report flags such as --format, --output, --pretty, and --include-output work with either target style.

Load and execute in Elixir

Use Aludel.Evals.FileSuite from releases, scripts, or application code:

alias Aludel.Evals.FileSuite

with {:ok, file_suite} <- FileSuite.load("evals/support-answer.yaml"),
     {:ok, suite_run} <- FileSuite.execute(file_suite) do
  {:ok, suite_run}
end

For the common one-step form:

{:ok, suite_run} =
  Aludel.Evals.FileSuite.load_and_execute("evals/support-answer.json")

In-memory callers can use load_string/2 with :json or :yaml before calling execute/1.

Validation and ownership

Manifest validation happens before database lookup or model execution. Aludel rejects:

• files larger than 256 KiB or content that is not valid UTF-8
• extensions other than .json, .yaml, and .yml
• malformed content, duplicate mapping keys, YAML aliases, explicit YAML tags, and multiple YAML documents
• missing or unknown fields and unsupported schema versions
• non-UUID target identifiers
• invalid sample counts, reducers, or minimum pass rates
• prompt versions that do not belong to the suite's prompt

Errors return a stable map with code and message fields. The Mix task wraps that error in its schema-version-2 JSON error envelope.

A manifest deliberately does not contain test cases, provider credentials, output paths, or quality-policy definitions. Editing or executing one cannot repopulate a suite or alter its dataset links. Manage those records in Aludel, then commit only their stable identifiers and execution settings.

CI example

- name: Run evaluation gate
  run: mix aludel.eval --file evals/support-answer.yaml --format github

Provider configuration and credentials should come from the application's normal runtime configuration. Do not commit secrets to a suite manifest.