mix test.json (ex_unit_json v0.5.0)

Copy Markdown View Source

Runs tests and outputs results as JSON.

This task wraps mix test and configures ExUnit to use ExUnitJSON.Formatter for JSON output instead of the default CLI formatter.

Setup

Add this to your project's mix.exs to ensure test.json runs in the test environment:

def cli do
  [preferred_envs: ["test.json": :test]]
end

This is required because Mix doesn't inherit preferred_envs from dependencies.

Usage

mix test.json
mix test.json test/my_test.exs
mix test.json test/my_test.exs:42

Options

All standard mix test options are supported, plus:

  • --summary-only - Output only the summary, omit individual test results
  • --all - Output all tests (default shows only failures)
  • --failures-only - Output only failed tests (DEFAULT - AI-optimized)
  • --first-failure - Output only the first failed test (quick iteration)
  • --filter-out PATTERN - Mark failures matching pattern as filtered (can repeat)
  • --output FILE - Write JSON to file instead of stdout
  • --compact - JSONL output with minimal fields (one line per test)
  • --group-by-error - Group failures by similar error message
  • --quiet - Suppress Logger output and TIP warnings for clean JSON piping
  • --no-warn - Suppress the "use --failed" warning when previous failures exist
  • --no-retry - Disable automatic retry of failed tests (see "Automatic Retry")
  • --cover - Enable code coverage (off by default for faster runs)
  • --cover-threshold N - Fail if overall coverage is below N (0-100). Requires --cover

Coverage

Coverage is disabled by default for faster test runs. Use --cover to enable:

mix test.json --cover

The JSON output includes a coverage key with:

"coverage": {
  "total_percentage": 96.96,
  "total_lines": 330,
  "covered_lines": 320,
  "threshold": 80,
  "threshold_met": true,
  "modules": [
    {
      "module": "MyApp.Module",
      "file": "lib/my_app/module.ex",
      "percentage": 92.68,
      "covered_lines": 38,
      "uncovered_lines": [45, 67, 89]
    }
  ]
}

Default Behavior (v0.3.0+)

By default, mix test.json outputs only failed tests (equivalent to --failures-only). This is optimized for AI agents where passing tests are noise.

Use --all to include all tests in output when needed.

Flag Precedence

When multiple filtering flags are combined, they follow this priority:

  1. --summary-only - Highest priority, omits tests array entirely
  2. --first-failure - Returns only the first failed test
  3. --failures-only / --all - Filter to failures or show all tests

For example, --summary-only --all will omit the tests array.

Iteration Workflow

When previous test failures exist, a tip is shown suggesting to use --failed for faster iteration:

TIP: 3 previous failure(s) exist. Consider:
  mix test.json --failed

This warning is skipped when:

  • --failed is already used
  • A specific file or directory is targeted
  • --only or --exclude tag filters are used
  • --no-warn flag is passed

Automatic Retry (v0.5.0+)

By default, when a run has failures, mix test.json automatically re-runs only the previously-failed tests (ExUnit's native --failed) in a subprocess, then merges the two runs to distinguish:

  • confirmed failures — failed both runs. Stay in tests, stay red.
  • flaky failures — failed run 1, passed run 2. Surfaced in a top-level flaky array (never hidden) but no longer block the run.

When every first-run failure heals on retry, the suite reports summary.result == "passed" and exits 0, so an AI agent isn't blocked by a flake — while the flaky tests are still named in the output. A test that fails both runs stays a hard failure (exit non-zero).

The merged output adds (only when a retry ran): a flaky array, a summary.flaky count, and a retry metadata block. The schema version stays 1 (additive).

Auto-retry is skipped (run-1 output is reported unchanged) when it would be meaningless or unsupported: --no-retry, config :ex_unit_json, retry: false, --failed (already iterating), --summary-only, --first-failure, --compact, --group-by-error, --filter-out, a file:line target, or an umbrella project.

Disable it entirely:

mix test.json --no-retry

# or in config/test.exs
config :ex_unit_json, retry: false

Strict Enforcement

To block full test runs when failures exist (useful for AI-assisted workflows):

# config/test.exs
config :ex_unit_json, enforce_failed: true

This will exit with an error instead of just warning.

Examples

# Run all tests with JSON output
mix test.json

# Run specific file
mix test.json test/my_test.exs

# Output only failures to a file
mix test.json --failures-only --output failures.json