Reports

The exit code says a run failed, not what failed. A caller that wants to route on the result asks for a report rather than scraping the console or re-running the tool.

mix quality --report .quality.json   # human output on stdout, report to a file
mix quality --format json            # report on stdout, human output on stderr
mix quality --report -               # the same, spelled the way a pipe reads

Both of the first two can be given at once. --report PATH is usually the more useful, because it leaves the human stream intact. --report - is --format json under another name: the report takes stdout and the human output moves to stderr, rather than the two being interleaved into something nothing can parse.

The report is built from the same results the human output is rendered from, so the two can never disagree.

Top level

{
  "status": "error",
  "version": "0.6.0",
  "duration_ms": 5014,
  "profile": "loop",
  "scope": "changed",
  "base_ref": "origin/main",
  "stages": []
}

status alone does not say what a run is evidence for. A green run scoped to three test files and a green full run are different claims, so anything that ratchets a recorded number, moves a baseline, or gates a merge has to check scope and refuse to move on anything but "all". That is what the field is for.

scope is the scope the run achieved. A :changed run that resolved to no test files ran the whole suite, so it reports "all" and puts the request on the Tests stage as requested_scope with a fallback_reason beside it. A caller checking scope == "all" never has to reason about fallbacks.

Stage

{
  "name": "Dependencies",
  "status": "error",
  "summary": "1 vulnerability (1 moderate)",
  "stats": {"vulnerabilities": 1, "vulnerabilities_by_severity": {"moderate": 1}},
  "findings": [],
  "duration_ms": 2400
}

Every stage carries the same keys whatever its status, so a consumer reads one field for the explanation rather than branching. A skipped stage puts its reason in summary:

{
  "name": "Dialyzer",
  "status": "skipped",
  "summary": "--quick",
  "stats": {},
  "findings": [],
  "duration_ms": 0
}

Reasons are the flag (--quick, --skip-credo, --skip <key>), the missing package (:gettext not installed), disabled in .quality.exs, compile failed, or - for a custom command declaring skip_exit_code - whatever the command itself said.

Every stage the run considered appears, skipped ones included, so absence is never something a caller has to interpret.

The Tests stage

The Tests stage carries what it ran as well as what it found, because a test count says nothing about how much of the suite produced it:

{
  "name": "Tests",
  "status": "ok",
  "summary": "12 of 12 passed (scope changed, 3 files vs origin/main, no coverage)",
  "scope": "changed",
  "files": 3,
  "test_files": [
    "test/user_test.exs",
    "test/user/email_test.exs",
    "test/accounts_test.exs"
  ],
  "base_ref": "origin/main",
  "coverage": "skipped",
  "coverage_reason": "not measured on a scoped run",
  "stats": {"test_count": 12, "passed_count": 12, "failed_count": 0},
  "findings": [],
  "duration_ms": 2800
}

A scoped run never reports a coverage percentage in stats. See Test scope.

Custom stages

A project's custom stages appear in stages[] like any other, so name is not one of a fixed nine and a consumer must not switch on it exhaustively. Route on status and findings, and treat name as a label.

A custom stage's stats are whatever its command reported, so the keys are its own rather than drawn from the built-in set. See configuration.md.

Finding

{
  "file": "lib/user.ex",
  "line": 42,
  "column": 3,
  "app": "web",
  "severity": "info",
  "check": "Credo.Check.Readability.ModuleDoc",
  "message": "Modules should have a @moduledoc tag."
}

Findings and output

findings is the parsed form; output is the fallback. For a failure, one of the two is always present.

A stage whose parser did not account for its tool's output carries that output verbatim under output instead of dropping it. Findings never replace output they did not account for, so a parser being wrong about a format degrades into "shows the original lines" rather than silently hiding a real problem.

Routing on a report

Read the report rather than re-running a tool to find out what broke.

report = "quality.json" |> File.read!() |> Jason.decode!()

for stage <- report["stages"], stage["status"] == "error" do
  case stage["findings"] do
    [] -> {stage["name"], {:output, stage["output"]}}
    findings -> {stage["name"], {:findings, findings}}
  end
end