Finds conformance tests that pass but are not yet in the regression registry, and adds them once they are verified.

Usage

# Report what could be ratcheted in, without writing anything
mix test.baseline

# Report, then add everything that passed
mix test.baseline --add

# Verify specific files and add them
mix test.baseline add test/scion_tests/basic/basic0_test.exs

# An authored case is named by its JSON file
mix test.baseline add conformance/cases/send/registered_immediate.json

Options

• --add - ratchet in every newly passing test the scan found
• --only - restrict the scan to one suite, scion, w3c or statifier
• --registry - registry to update, defaults to test/passing_tests.json

Both forms run each candidate on its own before writing anything, so a test can only enter the registry by passing. add is all-or-nothing: if any named file fails, the registry is left untouched.

The statifier suite is the cases this repository authors under conformance/cases/ (ADR-0070 decision 5). They have no generated test module, so a candidate there is the case's JSON file, and it is run through Mix.Statifier.Corpus.Runner, as mix statifier.corpus runs it, instead of mix test.

The ratchet only moves forward. Nothing here removes an entry - a test that used to pass and now does not is a regression to fix, not a line to delete.

A scan also prints a per-corpus coverage block: for each suite the scan covered, passing/total (percent%) against the suite's emitted corpus files. The numerator is every registry-tracked file plus whatever this scan found newly passing - tracked files this invocation skipped re-running are still counted, because mix test.regression is what guarantees them. add prints no such block: it verifies named files and never scans, so it has no denominator in hand.