Scoria.Eval manages eval datasets, specs, runs, score evidence, campaigns,
and reviewer queue candidates for release proof.
Use this module when a Phoenix app needs fail-closed eval evidence around prompts, model outputs, online scoring candidates, or dataset promotion. The host app owns expected behavior, business truth, prompt meaning, and release policy values. Scoria records immutable eval specs, run lineage, score evidence, review candidates, and approval lineage that reviewers can inspect.
See guides/reviewer-verification.md for verification suite ordering and
guides/ownership-boundary.md for the host-owned policy boundary.
Summary
Functions
Adds an item to a dataset, checking its state.
Finalizes a successful shard and updates the parent campaign summary row.
Marks an eval run complete and persists final aggregate facts.
Creates a campaign, child target rows, child eval runs, and batch-enqueues worker jobs.
Creates a dataset with state :open.
Assigns version to "1" if not provided.
Creates a campaign parent row and its runtime-only target rows atomically.
Creates an eval run with resolved spec and dataset snapshot facts.
Creates an eval spec.
Dismisses one active review candidate without removing durable score evidence.
Executes one campaign target through the shared orchestrator-backed judge path.
Finalizes a failed shard and narrows campaign-wide fatal state to explicit failure classes.
Gets a single dataset.
Gets a single eval spec.
Returns one projected review candidate or nil.
Returns one projected review candidate only when it belongs to the asserted dashboard tenant.
Lists campaign targets in insertion order.
Returns the list of dataset items for a given dataset id.
Returns the list of datasets.
Lists recent eval runs for one asserted dashboard tenant with rendered score evidence preloaded.
Returns the list of current eval specs.
Lists projected review-queue candidates for the operator UI.
Lists projected review-queue candidates visible to one asserted dashboard tenant.
Resolves the authoritative campaign/target/run lineage for a worker envelope. Persisted lineage remains the durable tenant truth even when envelope tenant data differs.
Moves a pending shard to running and refreshes aggregate campaign counters.
Builds a frozen preview for workflow-source promotion into a dataset item.
Promotes one review candidate into an open dataset and records durable queue lineage.
Promotes a Trace and its Spans into a new Dataset snapshot.
Promotes one original or replay workflow source into an existing dataset.
Persists per-item eval score evidence and updates aggregate run counters.
Replaces prior score truth for an eval run, keeping worker retries idempotent.
Requests sealed-baseline approval from one review candidate and keeps approval lineage visible.
Schedules online scoring sampling for a persisted trace on an async boundary.
Seals a dataset, making it immutable.
Returns summary strip counts for the projected review queue.
Returns summary strip counts for one asserted dashboard tenant's review queue.
Updates an eval spec immutably.
Functions
Adds an item to a dataset, checking its state.
Finalizes a successful shard and updates the parent campaign summary row.
Marks an eval run complete and persists final aggregate facts.
Creates a campaign, child target rows, child eval runs, and batch-enqueues worker jobs.
Creates a dataset with state :open.
Assigns version to "1" if not provided.
Creates a campaign parent row and its runtime-only target rows atomically.
Creates an eval run with resolved spec and dataset snapshot facts.
Creates an eval spec.
Dismisses one active review candidate without removing durable score evidence.
Executes one campaign target through the shared orchestrator-backed judge path.
Finalizes a failed shard and narrows campaign-wide fatal state to explicit failure classes.
Gets a single dataset.
Gets a single eval spec.
Returns one projected review candidate or nil.
Returns one projected review candidate only when it belongs to the asserted dashboard tenant.
Lists campaign targets in insertion order.
Returns the list of dataset items for a given dataset id.
Returns the list of datasets.
Lists recent eval runs for one asserted dashboard tenant with rendered score evidence preloaded.
Returns the list of current eval specs.
Lists projected review-queue candidates for the operator UI.
Lists projected review-queue candidates visible to one asserted dashboard tenant.
Resolves the authoritative campaign/target/run lineage for a worker envelope. Persisted lineage remains the durable tenant truth even when envelope tenant data differs.
Moves a pending shard to running and refreshes aggregate campaign counters.
Builds a frozen preview for workflow-source promotion into a dataset item.
Promotes one review candidate into an open dataset and records durable queue lineage.
Promotes a Trace and its Spans into a new Dataset snapshot.
Promotes one original or replay workflow source into an existing dataset.
Persists per-item eval score evidence and updates aggregate run counters.
Replaces prior score truth for an eval run, keeping worker retries idempotent.
Requests sealed-baseline approval from one review candidate and keeps approval lineage visible.
Schedules online scoring sampling for a persisted trace on an async boundary.
Seals a dataset, making it immutable.
Returns summary strip counts for the projected review queue.
Returns summary strip counts for one asserted dashboard tenant's review queue.
Updates an eval spec immutably.