mix corpus.coverage (predicator v6.0.0)

Copy Markdown View Source

Prints a checklist of behaviors the existing ExUnit suite exercises that the shipped conformance corpus does not (px-35i.4 Phase 6).

The corpus is authored JSON completed by the real pipeline, not extracted from the suite (see docs/plans/260807-px-35i.4-conformance-corpus.md's "The generation mechanism" section) - so nothing else tells an author what the 16k-line suite exercises that the corpus does not. This task closes that loop, using the existing suite as an authoring checklist rather than as the corpus's source of truth.

What it does

  1. Reads every test/**/*.exs file except test/predicator/conformance/** (the corpus's own test suite, not "the existing suite" this task checklists) and statically extracts literal Predicator.evaluate/2,3 and Predicator.compile/1 source strings via Predicator.Conformance.Coverage.extract_sources/1.
  2. Compiles each extracted source through the real compiler and counts opcode/operand patterns via Predicator.Conformance.Coverage.suite_pattern_frequencies/1.
  3. Reads the shipped conformance/corpus/tier-*.json cases and computes the same patterns via corpus_pattern_frequencies/1.
  4. Diffs the two, keeping every pattern the suite hits more than the corpus.
  5. Classifies each tier-5 (call:<name>) gap against the real builtin registry (Predicator.Context.new().functions's keys) via Coverage.classify/2, and prints the result grouped by tier - px-q1f, so a test-registered function name (boom, double, ...) reads as "not a corpus candidate" rather than as an unauthored gap, and a non-deterministic builtin (Date.now, Math.random, same for the relative_date opcode) reads as a documented exclusion rather than a bare corpus: 0.

This is dev-only tooling: it never writes a conformance case, never touches conformance/cases/, and never fails the gate - a wide report is expected and is not itself a failure. It is a heuristic report (see Predicator.Conformance.Coverage's moduledoc for exactly what it can and cannot see); its top entries need a human read before being authored as cases, since some will still be Elixir-internal test scaffolding that classify/2's registry check does not catch (e.g. an opts-handling assertion that happens to compile) rather than genuine corpus gaps.

Usage

mix corpus.coverage