mix test.regression (Statifier v2.9.0)

Copy Markdown View Source

Runs exactly the tests listed in test/passing_tests.json.

Those tests passed once, so any failure here is a regression rather than a missing feature. The conformance suites are excluded from mix test by default; this task includes the tags its registry entries need.

The statifier_tests list names authored conformance cases by their JSON files under conformance/cases/ (ADR-0070 decision 5). They have no test module, so this task runs them through Mix.Statifier.Corpus.Runner - as mix statifier.corpus runs them - once mix test has finished, and any one that disagrees with its expectation is a regression too.

Usage

mix test.regression
mix test.regression --registry test/passing_tests.json

Options

  • --registry - registry to run, defaults to test/passing_tests.json

Failures print ExUnit's own output, unedited. Growing the registry is mix test.baseline's job - this task never writes to it.

A registry entry that matches no file on disk fails the run. Skipping it would silently shrink the ratchet, which is the one thing it exists to prevent.

A passing run also prints a per-corpus coverage block: for each conformance suite, ratcheted/total (percent%) against the suite's emitted corpus files. The numerator is exactly the registry entries this run verified - unlike mix test.baseline's scan, there are no newly-passing files to add in. A failing run prints no such block; ExUnit's own output is the whole story then.

Summary

Functions

execute(argv, opts \\ [])

Runs the ratchet and reports the outcome instead of halting.

test_env()

Environment for the spawned mix test.

Functions

execute(argv, opts \\ [])

@spec execute(argv :: [String.t()], opts :: keyword()) :: :ok | {:error, String.t()}

Runs the ratchet and reports the outcome instead of halting.

opts[:runner] replaces the mix test shell-out with a function of the argument list returning an exit status, opts[:case_runner] replaces running the authored cases with a function of their paths returning what Mix.Statifier.Corpus.Runner.run_paths/2 returns, and opts[:root] moves the corpus scan behind the coverage block, and the authored cases, to a fixture tree. All three exist so the tests can drive this without spawning a nested mix test or starting the session runtime.

test_env()

@spec test_env() :: [{String.t(), String.t()}]

Environment for the spawned mix test.

The ratchet runs concurrently with mix quality's own Tests stage, in the same working directory, over largely the same modules. Scratch directories (Statifier.TmpDir) are collision-proof by construction - root/0 always ends in a System.pid() segment, so the two runs cannot land on the same path even by accident. STATIFIER_TMP_ROOT is set here anyway, not for isolation but so the ratchet's pid-scoped tree lands under a recognizable tmp/regression/<pid>/ rather than an anonymous tmp/<pid>/ indistinguishable from the Tests stage's own.