Feature Guide

Copy Markdown View Source

Aludel is a Phoenix-native workbench for comparing LLM behavior, building regression suites, and tracing prompt quality across versions, providers, cost, and latency.

Dashboard

The dashboard combines recent and lifetime activity:

  • rolling 7-day and 30-day comparisons
  • total executions, pass rate, weighted quality, total cost, cost per run, average latency, P50, and P95
  • outcome-weighted cost and latency efficiency
  • pass-rate stability and regression signals
  • interactive activity history
  • cost breakdowns by provider or prompt
  • latency and pass-rate breakdowns
  • recent evaluations and quick actions

Rolling metrics use complete evaluation history inside the selected window. Lifetime cards remain available as context when recent traffic is sparse.

Prompts and projects

Prompts are stable records with immutable template versions. Aludel extracts {{variable}} placeholders automatically and asks for values when a run starts.

Prompt management includes:

  • names, descriptions, tags, search, tag filters, and pagination
  • typed prompt projects with create, rename, filter, expand, and delete workflows
  • immutable prompt versions and a version-history rail
  • side-by-side template diffs
  • links from every prompt to run and evolution workflows

Suite projects are separate from prompt projects so each catalog can be organized independently.

Providers and models

Aludel supports OpenAI, Anthropic, Google Gemini, Ollama, xAI, Groq, and OpenRouter.

Provider forms:

  • discover active text-generation and chat models from LLMDB
  • retain deprecated models for existing configurations
  • accept custom model IDs
  • store JSON generation configuration for temperature and output limits
  • use built-in per-model token pricing or custom input/output rates

Provider credentials come from application configuration or environment variables and are not stored in provider rows. Changeset validation, a database constraint, and execution-boundary sanitization enforce this rule while preserving ordinary generation settings. Ollama does not require an API key and resolves to free token pricing by default.

Runs

A run applies one prompt version and one variable set to one or more providers. Multi-provider runs execute concurrently by default and can be configured for sequential dispatch.

Each provider result tracks:

  • pending, running, completed, or error state
  • raw output and a parsed JSON representation when possible
  • input/output tokens, latency, estimated cost, and callback metadata
  • a normalized execution artifact describing mode, inputs, documents, output, metrics, and bounded errors

The run page receives live status updates, handles partial provider failure, supports copying output or errors, and exports individual results as JSON.

Evaluation suites

Suites bind repeatable test cases to a prompt. Test cases can use template variables, multi-turn messages, document attachments, and one or more assertions.

The suite UI supports:

  • visual and raw JSON assertion editors
  • inline suite metadata and test-case editing
  • CSV and JSON imports with preview and row-level validation
  • population from reusable datasets
  • selection of a prompt version and provider for each suite run
  • persisted suite-run history with aggregate pass/fail, quality score, cost, and latency
  • detailed assertion, field-comparison, metric-context, and evaluator execution results
  • custom rubric judges and versioned correctness, relevance, faithfulness, safety, refusal, PII, and hallucination templates
  • bounded repeated sampling with all, any, majority, and minimum pass-rate reducers
  • validated quality-policy authoring with immutable version history for overall and metadata-group pass rates, evaluator scores, total cost, and average latency
  • policy version, aggregate status, and per-rule evidence on historical suite results
  • retrying one test result without rerunning the entire suite
  • copy actions, raw JSON exports, and previewable console, JSON v2, JUnit XML, and GitHub annotation reports

Built-in metrics are contains, not_contains, resource-bounded regex, exact_match, json_field, json_deep_compare, and rubric_judge. Regex patterns are validated before persistence and bounded again during execution, including for direct API and ExUnit calls. See the Evaluation Guide for examples and programmatic policy configuration.

The visual assertion editor configures either a built-in judge template or a custom rubric, a separate judge provider, a 0–100 pass threshold, an optional reference answer, and optional grounding context. The raw JSON editor exposes the same assertion contract. Once saved, judge assertions run through the dashboard, Mix CLI, ExUnit, file-based suites, and the Elixir API.

The suite run panel configures one to 20 attempts per test case and all, any, strict-majority, or minimum pass-rate reduction. Sampled results expose the aggregate pass evidence and an expandable ordered attempt history. File-based suites provide the same settings to the Mix CLI, while ExUnit and the Elixir API accept them as execution options.

From a suite page, Manage policy opens a JSON editor with live validation, a starter definition, rule guidance, and inspectable immutable history. The latest version applies to future runs. Historical result cards retain the version they used and display each rule's measured evidence. The dashboard and Elixir API create versions; the Mix CLI, ExUnit, reporters, and exports consume the resulting gate.

Each suite result links to a report workspace for safe browser previews and downloads. Console and GitHub formats omit generated responses, JSON includes them as part of its stable interchange schema, and JUnit requires explicit output opt-in. The dashboard bounds previews while keeping downloads complete and uncached. The same formats are available through mix aludel.eval and Aludel.Evals.Reporter.

Reusable datasets

Datasets are ordered collections of evaluation examples that can be reused by multiple suites. A dataset entry can contain:

  • a name and explicit position
  • prompt variable values
  • single-turn or multi-turn messages
  • assertions
  • arbitrary JSON metadata

Dataset pages support create, edit, delete, entry management, and JSON containment filters over metadata. Populating a suite copies entries in order, records source provenance, and skips entries already imported from that dataset.

Programmatic dataset workflows use Aludel.Datasets; its API covers ordered entry creation, metadata-filtered listing, provenance-preserving suite population, and safe repeat population.

Aludel.RedTeam provides seven versioned adversarial cases covering direct and indirect prompt injection, system prompt leakage, sensitive information disclosure, excessive agency, misinformation, and unsafe assistance. The dashboard and library API materialize selected cases into a reusable dataset with deterministic canary assertions, optional rubric judges, and category, severity, provenance, checksum, and deduplication metadata. The dashboard catalog exposes every prompt and risk field before selection. Matching reruns are idempotent; content or judge-configuration drift under the same key fails explicitly. See the red-team guide.

The dashboard and API can generate product-specific cases through an Aludel provider. Generation validates a strict response schema, bounds calls and output, reports sanitized partial failures and usage, and returns inert checksummed candidates without database writes or execution. Review exposes the complete receipt and leaves every candidate unapproved. A separate import action requires explicit approved candidate IDs, revalidates the review record, and creates the selected entries atomically with rubric judges and provenance.

Materialized and approved generated entries use the normal dataset workflow in the dashboard. Once they have been copied into a suite, that suite can be executed through the dashboard, mix aludel.eval, ExUnit, or Aludel.Evals. Catalog materialization and generated-case review and approval/import are dashboard and library API features. There is no separate red-team Mix command.

Documents and storage

Suite test cases accept PDF, PNG, JPEG, JSON, CSV, and plain-text attachments. File signatures and content are validated before persistence.

Document bytes are kept outside PostgreSQL through Aludel.Storage; database rows retain metadata and storage references. Included adapters cover:

  • local filesystem storage for development
  • AWS S3
  • Google Cloud Storage, including requester-pays buckets

The standalone release selects local, AWS S3, or GCS storage through validated environment settings. AWS can use its runtime identity provider or an explicit access-key pair, and local production requires an explicit persistent path.

Embedded applications use Aludel.Storage to create stable keys and read, write, or remove objects through the active or persisted historical backend.

Anthropic can receive PDFs natively. Providers that require images can use the configurable ImageMagick PDF converter.

Prompt evolution and optimization

Evolution analysis derives version-level and provider-level metrics from suite history:

  • pass rate and structured-output score
  • average and exact aggregate cost and latency
  • cost and latency per passed test
  • version-over-version deltas
  • pass-rate standard deviation and stability sample size
  • bounded regression, improvement, and insufficient-data signals
  • suite-scoped Pareto frontiers across quality, cost, and latency

The failure-reflection workflow uses failed suite evidence to request a variable-preserving prompt suggestion from a selected provider. Suggestions remain pending until a user explicitly accepts or dismisses them. Acceptance creates a new immutable prompt version and preserves the decision trail.

Exports and CI

Aludel provides:

  • JSON exports for individual run results
  • JSON exports for suite runs, including assertions, retries, callback metadata, and artifacts
  • JSON and CSV exports for prompt evolution metrics and provider breakdowns
  • Aludel.ExUnit assertions for inline generated output, existing suite runs, and execute-and-persist quality gates
  • schema-versioned JSON and YAML execution manifests that reference persisted suites without duplicating dataset ownership
  • Aludel.Evals.Reporter with console, schema-version-2 JSON, JUnit XML, GitHub annotation, and custom reporter support
  • mix aludel.eval for identifier-based or manifest-based headless suite execution and optional report file output

The Mix task accepts either --file PATH or the three target ID flags. It emits JSON by default and accepts --format console|json|junit|github, --output PATH, JSON-only --pretty, and JUnit-only --include-output. It exits unsuccessfully for invalid manifests or targets, execution errors, empty suites, or a non-passing active quality gate. The normalized report model keeps each output format independent from suite-run persistence.

See the file-based suite guide for manifest examples, the ExUnit evaluation guide for application-test examples, and the reporter guide for output formats, CI configuration, and custom reporter modules.

Execution modes

Native mode renders the prompt and calls the selected provider adapter. Callback mode delegates to a host module implementing Aludel.Executor, allowing evaluations to exercise retrieval, tools, routing, retries, or post-processing from the real application.

Both modes use the same run and suite UI. Callback responses require only output; tokens, latency, cost, and metadata are optional.

Direct library callers use Aludel.Execution.execute/1 for the normalized result contract or Aludel.Execution.execute_with_artifacts/1 when failure artifacts must be retained.

Embedding, access, and deployment

Aludel can be mounted inside a Phoenix router, run from the standalone application, or started with Docker Compose.

The embedded dashboard supports:

  • custom route names and instance names
  • custom auth/access resolvers and additional on_mount hooks
  • full-access or read-only operation
  • configurable refresh interval, LiveView socket path, and websocket/longpoll transport
  • custom logo links and CSP nonce assign keys
  • self-contained versioned CSS and JavaScript plus packaged fonts, icons, and images
  • light, dark, and system themes

The standalone release requires HTTP Basic Authentication in production and supports server-enforced read-only mode. Local development remains unauthenticated on loopback. See the Embedding Guide for configuration and TLS guidance.

Demo data

mix aludel.seed creates deterministic development data and is disabled in production. The catalog includes prompts, 14 configurations across all seven providers, datasets, suites, individual runs, suite runs, structured artifacts, representative failures, and 60 days of analytics history.

Running the task again refreshes the deterministic demo catalog instead of producing unrelated random examples.

Extension points

The public boundaries support host-specific integrations:

For operational examples, continue with the Evaluation Guide and Embedding Guide.