GitHub Actions integration

Tribunal works as an ordinary ExUnit suite for hard requirements and as a gated Mix batch for benchmark policy.

ExUnit safety checks

Generated tribunal_dataset tests and user-owned tribunal_assert tests run through mix test:

Tag user-owned suites so the filtered command includes them:

use ExUnit.Case
use Tribunal.ExUnit

@moduletag :eval

name: LLM safety

on: [pull_request]

jobs:
  safety:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: erlef/setup-beam@v1
        with:
          elixir-version: '1.18'
          otp-version: '27'
      - run: mix deps.get
      - run: mix test --only eval
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

ExUnit treats quality failures as assertion failures and provider, judge, or execution failures as errors.

Gated batch evaluation

Use a checked-in policy for repeatable CI settings:

# config/evaluation_policy.yaml
version: 1
datasets:
  - test/evals/benchmark.yaml
sampling:
  repeat: 3
  pass_rule: majority
gates:
  overall:
    threshold: 0.9
  groups:
    by: category
    threshold: 0.8

name: LLM evaluation

on:
  pull_request:
  workflow_dispatch:

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: erlef/setup-beam@v1
        with:
          elixir-version: '1.18'
          otp-version: '27'
      - run: mix deps.get
      - name: Run evaluation gate
        run: mix tribunal.eval --config config/evaluation_policy.yaml --format github
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

Explicit CLI flags override the policy, and positional dataset files replace its dataset list:

mix tribunal.eval test/evals/pull_request.yaml \
  --config config/evaluation_policy.yaml \
  --repeat 5 \
  --pass-rule rate:0.8 \
  --threshold 0.85 \
  --group-by category \
  --group-threshold 0.75

Policy values override defaults. Dataset paths are resolved from the repository working directory.

Exit behavior

Without an overall or group quality gate, configured through CLI flags or policy, ordinary quality failures are report-only. The command exits zero with gate_status: not_configured.

These conditions always exit nonzero:

• a configured overall or group gate fails
• a provider, assertion, judge, task, or timeout produces an operational error
• configuration is invalid
• the selected dataset contains zero cases

Operational errors set gate_status: error. They cannot be hidden by any, majority, or a rate sampling rule. An ungated operational error keeps threshold_passed: null because no quality gate was configured.

--strict is a zero-tolerance overall gate and cannot be combined with --threshold. --group-by and --group-threshold must be supplied together.

Saving reports even when the gate fails

A failing command stops later shell commands in the same step. Generate the machine report in a step that records the exit code, upload it with if: always(), then preserve the original status:

- name: Run evaluation
  id: tribunal
  continue-on-error: true
  run: >-
    mix tribunal.eval
    --config config/evaluation_policy.yaml
    --format json
    --output results.json

- name: Upload evaluation report
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: tribunal-${{ github.sha }}
    path: results.json

- name: Enforce evaluation result
  if: steps.tribunal.outcome == 'failure'
  run: exit 1

JSON reports use schema version 3 and include reduced case totals, attempt totals, ordered attempt evidence, sampling summaries, and overall and group gate results.

JUnit reporting

JUnit distinguishes assertion failures from operational errors:

- name: Produce JUnit report
  id: tribunal_junit
  continue-on-error: true
  run: mix tribunal.eval --config config/evaluation_policy.yaml --format junit --output junit.xml

- name: Publish JUnit report
  if: always()
  uses: mikepenz/action-junit-report@v4
  with:
    report_paths: junit.xml

- name: Enforce evaluation result
  if: steps.tribunal_junit.outcome == 'failure'
  run: exit 1

Scheduled baselines

For broad report-only tracking, omit quality gates but keep operational failures blocking:

name: Scheduled LLM baseline

on:
  schedule:
    - cron: '0 6 * * *'
  workflow_dispatch:

jobs:
  baseline:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: erlef/setup-beam@v1
        with:
          elixir-version: '1.18'
          otp-version: '27'
      - run: mix deps.get
      - run: >-
          mix tribunal.eval test/evals/benchmark.yaml
          --repeat 3
          --pass-rule majority
          --format json
          --output results.json
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: tribunal-baseline-${{ github.run_id }}
          path: results.json
          retention-days: 90

GitHub dependency caching is independent from Tribunal. Tribunal does not currently provide provider-output caching, assertion-result caching, resume, or infrastructure retries.