Aludel.Evals.FileSuite (aludel v0.7.0)

Loads and executes versioned JSON or YAML suite manifests.

A manifest identifies a persisted suite, prompt version, and provider. It may also define bounded repeated-sampling settings. Test cases and dataset provenance remain owned by the database; executing a manifest never imports, replaces, or deletes suite data.

Schema version 1 accepts this shape:

schema_version: 1
suite_id: 9a756a58-eaec-43ca-99e6-f5c016d85d0c
prompt_version_id: e74cf2e1-94b6-4bcb-9ed9-b259661be906
provider_id: e1c60ec0-6d55-419b-b958-7d088055254f
sampling:
  samples: 5
  reducer: majority

Supported reducers are all, any, majority, and minimum_pass_rate. The latter also requires minimum_pass_rate between 0.0 and 1.0.

Files are limited to 256 KiB. Parsing rejects unsupported extensions, invalid UTF-8, aliases, explicit YAML tags, multiple YAML documents, duplicate mapping keys, unknown fields, and unsupported schema versions before database access.

Summary

Types

error()

format()

t()

Functions

execute(file_suite)

Executes a validated manifest against its persisted suite.

load(path)

Loads and validates a .json, .yaml, or .yml manifest.

load_and_execute(path)

Loads, validates, and executes a suite manifest.

load_string(contents, format)

Parses and validates manifest content in :json or :yaml format.

Types

error()

@type error() :: %{code: String.t(), message: String.t()}

format()

@type format() :: :json | :yaml

t()

@type t() :: %Aludel.Evals.FileSuite{
  prompt_version_id: Ecto.UUID.t(),
  provider_id: Ecto.UUID.t(),
  sampling: Aludel.Evals.Sampling.t(),
  schema_version: 1,
  suite_id: Ecto.UUID.t()
}

Functions

execute(file_suite)

@spec execute(t()) :: {:ok, Aludel.Evals.SuiteRun.t()} | {:error, error()}

Executes a validated manifest against its persisted suite.

The prompt version must belong to the suite's prompt. The suite's existing test cases, document associations, and latest quality policy are used without modifying dataset ownership.

load(path)

@spec load(Path.t()) :: {:ok, t()} | {:error, error()}

Loads and validates a .json, .yaml, or .yml manifest.

The file is read with a fixed upper bound. Errors have stable code and message fields suitable for command-line and application handling.

load_and_execute(path)

@spec load_and_execute(Path.t()) ::
  {:ok, Aludel.Evals.SuiteRun.t()} | {:error, error()}

Loads, validates, and executes a suite manifest.

Invalid files and targets return stable errors instead of starting a partial evaluation.

load_string(contents, format)

@spec load_string(binary(), format()) :: {:ok, t()} | {:error, error()}

Parses and validates manifest content in :json or :yaml format.

The same size, encoding, duplicate-key, schema, identifier, and sampling checks used by load/1 apply to in-memory content.