Modules
LLM evaluation framework for Elixir.
Assertion evaluation engine.
Deterministic assertions that don't require LLM calls.
Embedding-based semantic similarity assertions.
LLM-as-judge assertions for evaluating LLM outputs.
Loads evaluation datasets from JSON or YAML files.
ExUnit integration for LLM evaluations.
ExUnit-style assertion macros for LLM evaluation.
Behaviour for LLM-as-judge assertions.
Detects stereotypes and prejudice in LLM outputs.
Compares LLM output against an expected answer.
Detects whether an LLM-powered assistant claimed to have performed (or committed to performing) an action outside its actual capabilities.
Evaluates whether LLM output is grounded in provided context.
Detects whether an LLM-powered assistant fabricated specifics in its response.
Detects claims not supported by the provided context.
Detects dangerous or harmful content in LLM outputs.
Detects whether an LLM-powered assistant was hijacked off-topic.
Detects whether an LLM-powered assistant impersonated a person, role, or authority it shouldn't be speaking as.
Detects when an LLM has been manipulated to bypass safety measures.
Detects Personally Identifiable Information (PII) in LLM outputs.
Detects whether an LLM response violates a supplied policy.
Detects whether an LLM-powered assistant leaked its system prompt or internal configuration in its response.
Detects when an LLM appropriately refuses to comply with a request.
Evaluates whether LLM output is relevant to the input query.
Detects hostile, abusive, or toxic content in LLM outputs.
Red team attack generators for testing LLM safety.
Behaviour for the LLM that generates adversarial prompts.
Default attacker LLM, backed by req_llm.
Stub attacker for tests. Returns canned responses set via set_response/1.
Behaviour for red-team attack generators.
Shared implementation for the built-in LLM-driven red-team plugins.
Generates attacks that try to get the assistant to claim it can perform actions outside its actual capabilities.
Generates attacks that probe for hallucination — fabricated specifics the assistant has no way of actually knowing.
Generates off-topic-but-adjacent attacks that try to hijack the assistant.
Generates attacks that try to get the assistant to impersonate a brand, staff member, or authority figure.
Generates attacks targeting policy violations.
Generates attacks that try to leak the assistant's system prompt or internal instructions.
Emits a list of red-team cases as a tribunal-shaped dataset YAML.
Behaviour for eval result reporters.
Pretty console output for eval results.
GitHub Actions annotations format.
HTML report for shareable results.
JSON output for CI/machine consumption.
JUnit XML format for CI tools.
Plain ASCII text output (no unicode).
Represents a single evaluation test case.
Mix Tasks
Runs LLM evaluations from dataset files.
Creates the eval directory structure with example files.
Generates a red-team dataset by invoking one or more plugins.