mix test.baseline (Statifier v2.9.0)

Finds conformance tests that pass but are not yet in the regression registry, and adds them once they are verified.

Usage

# Report what could be ratcheted in, without writing anything
mix test.baseline

# Report, then add everything that passed
mix test.baseline --add

# Verify specific files and add them
mix test.baseline add test/scion_tests/basic/basic0_test.exs

# An authored case is named by its JSON file
mix test.baseline add conformance/cases/send/registered_immediate.json

Options

--add - ratchet in every newly passing test the scan found
--only - restrict the scan to one suite, scion, w3c or statifier
--registry - registry to update, defaults to test/passing_tests.json

Both forms run each candidate on its own before writing anything, so a test can only enter the registry by passing. add is all-or-nothing: if any named file fails, the registry is left untouched.

The statifier suite is the cases this repository authors under conformance/cases/ (ADR-0070 decision 5). They have no generated test module, so a candidate there is the case's JSON file, and it is run through Mix.Statifier.Corpus.Runner, as mix statifier.corpus runs it, instead of mix test.

The ratchet only moves forward. Nothing here removes an entry - a test that used to pass and now does not is a regression to fix, not a line to delete.

A scan also prints a per-corpus coverage block: for each suite the scan covered, passing/total (percent%) against the suite's emitted corpus files. The numerator is every registry-tracked file plus whatever this scan found newly passing - tracked files this invocation skipped re-running are still counted, because mix test.regression is what guarantees them. add prints no such block: it verifies named files and never scans, so it has no denominator in hand.

Summary

Functions

execute(argv, opts \\ [])

Runs the task and reports the outcome instead of halting.

Functions

execute(argv, opts \\ [])

@spec execute(argv :: [String.t()], opts :: keyword()) :: :ok | {:error, String.t()}

Runs the task and reports the outcome instead of halting.

opts[:runner] replaces the mix test shell-out with a function of the argument list returning an exit status, opts[:case_runner] replaces running the authored cases with a function of their paths returning what Mix.Statifier.Corpus.Runner.run_paths/2 returns, opts[:root] moves the corpus scan to a fixture tree, and opts[:today] fixes the date stamped into the registry. All four exist so the tests can drive this without spawning a nested mix test or starting the session runtime.