Aludel.RedTeam (aludel v0.7.0)

Curated and generated adversarial evaluation cases.

Catalog cases can be inspected directly or materialized into an existing reusable dataset. Materialization is transactional and idempotent for the same case version and prompt variable. It records provenance, risk category, severity, content checksum, and a stable deduplication key in entry metadata.

Curated cases include deterministic canary assertions. Generation returns inert review values without database writes or execution; a separate import call requires explicit approved candidate IDs.

Summary

Types

category()

generate_error()

generated_import_error()

materialization()

materialize_error()

template()

Functions

all()

Returns every case in stable catalog order.

categories()

Returns the supported risk categories in alphabetical order.

fetch(id)

Fetches a catalog case by its stable ID.

fetch!(id)

Fetches a catalog case by its stable ID, raising KeyError when absent.

generate(provider_id, opts \\ [])

Generates bounded, reviewable adversarial cases without persisting them.

import_generated(dataset, generation, opts \\ [])

Imports explicitly approved generated cases into an existing dataset.

materialize(dataset, opts \\ [])

Materializes selected catalog cases into an existing dataset.

Types

category()

@type category() :: Aludel.RedTeam.Catalog.category()

generate_error()

@type generate_error() ::
  :provider_not_found
  | :invalid_options
  | :invalid_categories
  | :invalid_cases_per_category
  | :invalid_target_context
  | :invalid_budget
  | {:unknown_categories, [term()]}

generated_import_error()

@type generated_import_error() ::
  :dataset_not_found
  | :invalid_generation
  | :invalid_options
  | :invalid_approved_case_ids
  | :invalid_variable
  | :invalid_judge_provider_id
  | :invalid_judge_threshold
  | {:unknown_case_ids, [term()]}
  | {:deduplication_conflict, String.t()}
  | Ecto.Changeset.t()

materialization()

@type materialization() :: %{
  created: [Aludel.Datasets.DatasetEntry.t()],
  skipped: [Aludel.Datasets.DatasetEntry.t()]
}

materialize_error()

@type materialize_error() ::
  :dataset_not_found
  | :invalid_options
  | :invalid_variable
  | :invalid_judge_provider_id
  | :invalid_judge_threshold
  | {:unknown_categories, [term()]}
  | {:unknown_case_ids, [String.t()]}
  | {:deduplication_conflict, String.t()}
  | Ecto.Changeset.t()

template()

@type template() :: Aludel.RedTeam.Catalog.template()

Functions

all()

@spec all() :: [template()]

Returns every case in stable catalog order.

categories()

@spec categories() :: [category()]

Returns the supported risk categories in alphabetical order.

fetch(id)

@spec fetch(String.t()) :: {:ok, template()} | :error

Fetches a catalog case by its stable ID.

fetch!(id)

@spec fetch!(String.t()) :: template()

Fetches a catalog case by its stable ID, raising KeyError when absent.

generate(provider_id, opts \\ [])

@spec generate(
  String.t(),
  keyword()
) :: {:ok, Aludel.RedTeam.Generation.t()} | {:error, generate_error()}

Generates bounded, reviewable adversarial cases without persisting them.

Generation runs once per selected category and records sanitized failures, observed token and cost usage, and the configured limits. Successfully validated categories remain available when another category fails.

Options:

:categories - non-empty, unique category list; defaults to [:prompt_injection]
:cases_per_category - cases requested per category from 1 to 5; defaults to 2
:target_context - optional product or policy context, limited to 10,000 characters
:max_requests - category request limit from 1 to 6; defaults to 6
:max_output_tokens - per-request output limit from 100 to 4,000; defaults to 1,200
:max_total_tokens - observed token stop threshold up to 100,000; defaults to 20,000
:max_cost_usd - observed cost stop threshold greater than 0 and up to 100; defaults to 5
:request_timeout_ms - per-request timeout from 100 to 120,000; defaults to 30,000

See Aludel.RedTeam.Generation for the returned review boundary. Review the candidates and failures before deciding whether any case should enter an evaluation dataset.

import_generated(dataset, generation, opts \\ [])

@spec import_generated(
  Aludel.Datasets.Dataset.t(),
  Aludel.RedTeam.Generation.t(),
  keyword()
) ::
  {:ok, materialization()} | {:error, generated_import_error()}

Imports explicitly approved generated cases into an existing dataset.

The generation checksum and every approved case checksum are revalidated before the dataset is locked. The complete approved selection is written atomically and retains generation, review, usage, limit, and judge provenance.

Options:

:approved_case_ids - required non-empty unique list of candidate IDs
:variable - prompt variable populated by each case; defaults to "input"
:judge_provider_id - provider UUID for rubric judging; defaults to the generator provider
:judge_threshold - rubric judge pass threshold from 0 to 100; defaults to 80

Repeating an exact import skips its existing entries. Changed payload, provenance, review, or judge configuration under the same key returns a conflict and rolls back the entire selection.

materialize(dataset, opts \\ [])

@spec materialize(
  Aludel.Datasets.Dataset.t(),
  keyword()
) :: {:ok, materialization()} | {:error, materialize_error()}

Materializes selected catalog cases into an existing dataset.

Options:

:categories - category atoms to include; defaults to every category
:case_ids - stable case IDs to include; defaults to every case
:variable - prompt variable populated by each case; defaults to "input"
:judge_provider_id - optional provider UUID for a recommended rubric judge
:judge_threshold - rubric judge pass threshold from 0 to 100; defaults to 80

The two selectors are combined. Repeating a materialization with the same case version and variable skips the previously created entries. A checksum mismatch under the same deduplication key returns an error instead of silently accepting different content.