mix tribunal.eval (Tribunal v3.0.0)

Copy Markdown View Source

Runs LLM evaluations from dataset files.

Loading messages and live attempt progress go to stderr. The final report goes to stdout or the file selected with --output.

Usage

mix tribunal.eval [options] [files...]

Options

  • --config - Versioned YAML policy with datasets, sampling, and gates
  • --format - Output format: console (default), text, json, html, github, junit
  • --output - Write results to file instead of stdout
  • --provider - Module.function to call for each test case (e.g. MyApp.Agent.query)
  • --threshold - Minimum pass rate (0.0-1.0) required. Default: none (quality failures are report-only)
  • --strict - Fail on any failure, equivalent to --threshold 1.0 (for CI gates)
  • --concurrency - Number of test cases to run in parallel. Default: 1 (sequential)
  • --limit - Maximum number of test cases to evaluate
  • --offset - Number of test cases to skip before evaluating. Default: 0
  • --repeat - Number of fresh attempts per case. Default: 1
  • --pass-rule - Attempt rule: all, any, majority, or rate:0.8. Default: all

Provider Function

The provider function receives a Tribunal.TestCase struct and should return the LLM output as a string. The test case includes:

  • input - The query/prompt
  • context - Optional context for RAG-style queries
  • expected_output - Optional expected answer

Example provider:

def query(%Tribunal.TestCase{input: input, context: context}) do
  # Call your LLM here
  MyApp.LLM.generate(input, context: context)
end

Examples

# Run all evals in default location
mix tribunal.eval

# Run specific dataset
mix tribunal.eval test/evals/datasets/questions.json

# Run with a provider to generate outputs
mix tribunal.eval --provider MyApp.Agent.query

# Output JSON for CI
mix tribunal.eval --format json --output results.json

# GitHub Actions annotations
mix tribunal.eval --format github

# Default: complete without a quality gate (for baseline tracking)
mix tribunal.eval

# Fail if pass rate < 80%
mix tribunal.eval --threshold 0.8

# Strict mode: fail on any failure (for CI gates)
mix tribunal.eval --strict

# Run 5 test cases in parallel
mix tribunal.eval --concurrency 5

# Evaluate only the first 50 cases
mix tribunal.eval --limit 50

# Skip 30 cases, then evaluate the next 50
mix tribunal.eval --offset 30 --limit 50