Finds conformance tests that pass but are not yet in the regression registry, and adds them once they are verified.

Usage

# Report what could be ratcheted in, without writing anything
mix test.baseline

# Report, then add everything that passed
mix test.baseline --add

# Verify specific files and add them
mix test.baseline add test/scion_tests/basic/basic0_test.exs

# An authored case is named by its JSON file
mix test.baseline add conformance/cases/send/registered_immediate.json

Options

• --add - ratchet in every newly passing test the scan found
• --only - restrict the scan to one suite, scion, w3c or statifier
• --registry - registry to update, defaults to test/passing_tests.json

Both forms run each candidate on its own before writing anything, so a test can only enter the registry by passing. add is all-or-nothing: if any named file fails, the registry is left untouched.

The statifier suite is the cases this repository authors under conformance/cases/ (ADR-0070 decision 5). They have no generated test module, so a candidate there is the case's JSON file, and it is run through Mix.Statifier.Corpus.Runner, as mix statifier.corpus runs it, instead of mix test.

Running them needs OTP's :inets, :ssl and :public_key applications, which Mix.Statifier.Corpus.Runner.start_runtime/0 puts on the code path before the first authored case runs: on an OTP built without :ssl, a scan that finds an untracked statifier candidate, or an add that names an authored case, raises there.

The ratchet only moves forward. Nothing here removes an entry - a test that used to pass and now does not is a regression to fix, not a line to delete.

A scan also prints a per-corpus coverage block: for each suite the scan covered, passing/total (percent%) against the suite's emitted corpus files. The numerator is every registry-tracked file plus whatever this scan found newly passing - tracked files this invocation skipped re-running are still counted, because mix test.regression is what guarantees them. add prints no such block: it verifies named files and never scans, so it has no denominator in hand.