# Tribunal v1.4.0 - Table of Contents

> LLM evaluation framework for Elixir

## Pages

- [Tribunal ⚖️](readme.md)

- Guides
  - [Getting Started](getting-started.md)
  - [Test Mode vs Evaluation Mode](evaluation-modes.md)
  - [ExUnit Integration](exunit-integration.md)
  - [Assertions Reference](assertions.md)
  - [LLM-as-Judge](llm-as-judge.md)
  - [Datasets](datasets.md)
  - [Red Team Testing](red-team-testing.md)
  - [Reporters](reporters.md)
  - [GitHub Actions Integration](github-actions.md)

## Modules

- [Tribunal.EvalCase.Assertions](Tribunal.EvalCase.Assertions.md): ExUnit-style assertion macros for LLM evaluation.

- [Tribunal.Judges.PII](Tribunal.Judges.PII.md): Detects Personally Identifiable Information (PII) in LLM outputs.

- Core
  - [Tribunal](Tribunal.md): LLM evaluation framework for Elixir.
  - [Tribunal.Assertions](Tribunal.Assertions.md): Assertion evaluation engine.
  - [Tribunal.TestCase](Tribunal.TestCase.md): Represents a single evaluation test case.

- Assertion Types
  - [Tribunal.Assertions.Deterministic](Tribunal.Assertions.Deterministic.md): Deterministic assertions that don't require LLM calls.
  - [Tribunal.Assertions.Embedding](Tribunal.Assertions.Embedding.md): Embedding-based semantic similarity assertions.
  - [Tribunal.Assertions.Judge](Tribunal.Assertions.Judge.md): LLM-as-judge assertions for evaluating LLM outputs.

- Testing
  - [Tribunal.Dataset](Tribunal.Dataset.md): Loads evaluation datasets from JSON or YAML files.
  - [Tribunal.EvalCase](Tribunal.EvalCase.md): ExUnit integration for LLM evaluations.
  - [Tribunal.RedTeam](Tribunal.RedTeam.md): Red team attack generators for testing LLM safety.

- Judges
  - [Tribunal.Judge](Tribunal.Judge.md): Behaviour for LLM-as-judge assertions.
  - [Tribunal.Judges.Bias](Tribunal.Judges.Bias.md): Detects stereotypes and prejudice in LLM outputs.
  - [Tribunal.Judges.Correctness](Tribunal.Judges.Correctness.md): Compares LLM output against an expected answer.
  - [Tribunal.Judges.ExcessiveAgency](Tribunal.Judges.ExcessiveAgency.md): Detects whether an LLM-powered assistant claimed to have performed (or
committed to performing) an action outside its actual capabilities.
  - [Tribunal.Judges.Faithful](Tribunal.Judges.Faithful.md): Evaluates whether LLM output is grounded in provided context.
  - [Tribunal.Judges.Hallucinated](Tribunal.Judges.Hallucinated.md): Detects whether an LLM-powered assistant fabricated specifics in its
response.
  - [Tribunal.Judges.Hallucination](Tribunal.Judges.Hallucination.md): Detects claims not supported by the provided context.
  - [Tribunal.Judges.Harmful](Tribunal.Judges.Harmful.md): Detects dangerous or harmful content in LLM outputs.
  - [Tribunal.Judges.Hijacked](Tribunal.Judges.Hijacked.md): Detects whether an LLM-powered assistant was hijacked off-topic.
  - [Tribunal.Judges.Imitation](Tribunal.Judges.Imitation.md): Detects whether an LLM-powered assistant impersonated a person, role, or
authority it shouldn't be speaking as.
  - [Tribunal.Judges.Jailbreak](Tribunal.Judges.Jailbreak.md): Detects when an LLM has been manipulated to bypass safety measures.
  - [Tribunal.Judges.PolicyViolation](Tribunal.Judges.PolicyViolation.md): Detects whether an LLM response violates a supplied policy.
  - [Tribunal.Judges.PromptExtracted](Tribunal.Judges.PromptExtracted.md): Detects whether an LLM-powered assistant leaked its system prompt or
internal configuration in its response.
  - [Tribunal.Judges.Refusal](Tribunal.Judges.Refusal.md): Detects when an LLM appropriately refuses to comply with a request.
  - [Tribunal.Judges.Relevant](Tribunal.Judges.Relevant.md): Evaluates whether LLM output is relevant to the input query.
  - [Tribunal.Judges.Toxicity](Tribunal.Judges.Toxicity.md): Detects hostile, abusive, or toxic content in LLM outputs.

- Red Team
  - [Tribunal.RedTeam.Attacker](Tribunal.RedTeam.Attacker.md): Behaviour for the LLM that generates adversarial prompts.
  - [Tribunal.RedTeam.Attacker.ReqLLM](Tribunal.RedTeam.Attacker.ReqLLM.md): Default attacker LLM, backed by `req_llm`.
  - [Tribunal.RedTeam.Attacker.Stub](Tribunal.RedTeam.Attacker.Stub.md): Stub attacker for tests. Returns canned responses set via `set_response/1`.
  - [Tribunal.RedTeam.Plugin](Tribunal.RedTeam.Plugin.md): Behaviour for red-team attack generators.
  - [Tribunal.RedTeam.Plugin.Base](Tribunal.RedTeam.Plugin.Base.md): Shared implementation for the built-in LLM-driven red-team plugins.
  - [Tribunal.RedTeam.Plugins.ExcessiveAgency](Tribunal.RedTeam.Plugins.ExcessiveAgency.md): Generates attacks that try to get the assistant to claim it can perform
actions outside its actual capabilities.
  - [Tribunal.RedTeam.Plugins.Hallucination](Tribunal.RedTeam.Plugins.Hallucination.md): Generates attacks that probe for hallucination — fabricated specifics
the assistant has no way of actually knowing.
  - [Tribunal.RedTeam.Plugins.Hijacking](Tribunal.RedTeam.Plugins.Hijacking.md): Generates off-topic-but-adjacent attacks that try to hijack the assistant.
  - [Tribunal.RedTeam.Plugins.Imitation](Tribunal.RedTeam.Plugins.Imitation.md): Generates attacks that try to get the assistant to impersonate a brand,
staff member, or authority figure.
  - [Tribunal.RedTeam.Plugins.Policy](Tribunal.RedTeam.Plugins.Policy.md): Generates attacks targeting policy violations.
  - [Tribunal.RedTeam.Plugins.PromptExtraction](Tribunal.RedTeam.Plugins.PromptExtraction.md): Generates attacks that try to leak the assistant's system prompt or
internal instructions.
  - [Tribunal.RedTeam.YamlEmit](Tribunal.RedTeam.YamlEmit.md): Emits a list of red-team cases as a tribunal-shaped dataset YAML.

- Reporters
  - [Tribunal.Reporter](Tribunal.Reporter.md): Behaviour for eval result reporters.

  - [Tribunal.Reporter.Console](Tribunal.Reporter.Console.md): Pretty console output for eval results.

  - [Tribunal.Reporter.GitHub](Tribunal.Reporter.GitHub.md): GitHub Actions annotations format.

  - [Tribunal.Reporter.HTML](Tribunal.Reporter.HTML.md): HTML report for shareable results.

  - [Tribunal.Reporter.JSON](Tribunal.Reporter.JSON.md): JSON output for CI/machine consumption.

  - [Tribunal.Reporter.JUnit](Tribunal.Reporter.JUnit.md): JUnit XML format for CI tools.

  - [Tribunal.Reporter.Text](Tribunal.Reporter.Text.md): Plain ASCII text output (no unicode).

## Mix Tasks

- [mix tribunal.eval](Mix.Tasks.Tribunal.Eval.md): Runs LLM evaluations from dataset files.
- [mix tribunal.init](Mix.Tasks.Tribunal.Init.md): Creates the eval directory structure with example files.
- [mix tribunal.redteam.generate](Mix.Tasks.Tribunal.Redteam.Generate.md): Generates a red-team dataset by invoking one or more plugins.

