Mutare.Runner.CoverageProbe (mutare v0.3.1)

Copy Markdown View Source

Select the test files to run for each mutant — coverage-driven test selection.

Runs after Mutare.Runner.Baseline has confirmed the suite green and measured the timing; this module is purely about coverage and never green-checks or times anything. Keeping the two apart is the point: folding coverage into the baseline made baseline_ms a sum of per-file process boots (an inflated timeout cap), and meant the suite was never confirmed green together — only file-by-file.

It is one instrumented suite run. The probe runs mix test once at baseline with the coverage-capture flag set, so the metamutant self-records — in the test process, synchronously — which mutant ids each test file covers, plus a process-agnostic aggregate of every id that ran at all (Mutare.Coverage.Recorder implements the capture, Mutare.Coverage reads the dump). No :cover, no per-file subprocess fan-out, and no async-formatter race that loses fast async: false modules' coverage.

Three modes, set by :test_selection, a granularity ladder from safest/slowest to fastest — :full ⊃ :coverage ⊃ :tests:

  • :tests (default) — per mutant, run only the individual test cases that covered its line, via mix test <file> --only test:<name> (one --only per covering test). Finer than :coverage: where :coverage runs a whole covering file, :tests runs just the tests in it that touched the line. It falls back to :coverage's whole-file selection for any id whose coverage can't be pinned to runnable tests — an id in the wholefile set (covered through a setup_all/on_exit, which map to a module-scoped context, not a single test), or one covered via an unlabeled process (whole suite). So the narrowing is applied only where every covering attribution is a concrete, runnable test.
  • :coverage — per mutant, run only the test files that covered its line (a setup_all attributes to its own module's file via the __ex_unit__/2 stacktrace frame; a Task via its caller chain; a test-registered on_exit via its closure frame); a mutant whose code ran in an unlabeled process (a bare spawn, a setup-registered on_exit closure, or a setup_all whose work happened in a spawned Task) runs the whole suite — even if some file also attributes it, since that partial attribution would otherwise mask the unlabeled coverage and produce a false survivor; a mutant that never ran at all is :no_coverage (skipped, and kept out of the score's denominator).
  • :full — no per-file selection: every covered mutant runs the whole suite, the rest are :no_coverage. Safest for suites with cross-file dependencies — including a setup_all with cross-module global side effects, which :coverage attributes to its own file only (see Mutare.Coverage.Recorder).

:tests and the file-granular safety margin it trades away

:coverage is file-granular on purpose: if any test in a file covers the line, the whole file runs — so a sibling test that kills the mutant indirectly (fails because a covering test in the same async: false module ran the mutated line and left corrupt shared state, without the sibling touching the line itself) is still run. :tests narrows to the covering tests only, dropping that sibling — so on a stateful, cross-test-dependent suite :tests can turn such a kill into a false survivor. It is the default on the assumption that suites are predominantly async: true with per-test-isolated state (where a sibling that never runs the line cannot observe the mutation, so nothing is lost); :coverage is the opt-out (--per-file) and :full the fully conservative escape hatch.

The narrowing never manufactures a false kill: every name emitted is a test that actually ran in one of the included files during the probe, so at least one test always matches and mix test --only never hits its "no test executed" error (which would otherwise surface as a harness error, not a kill).

Coverage is advisory, never authoritative. Anything uncertain — a non-zero probe exit, an unreadable dump, or missing capture tables — degrades to :run_all: we never skip a mutant on doubt. A valid empty dump means none of the emitted mutants ran; focused selection can legitimately leave the entire aggregate empty. But because :run_all makes every covered mutant run the whole suite — prohibitive on a large project — a failed probe run is retried once before degrading: the baseline was green moments earlier, so a probe failure is usually a flaky test, and one extra suite run is cheap next to a whole run's selection quality. The probe run is wall-clock capped for the same reason (by default a generous multiple of the per-mutant cap, since instrumentation adds overhead a plain baseline doesn't have; :probe_timeout sets an explicit cap instead): a pathological interaction between the coverage capture and the target's hot loops must degrade to :run_all, not hang the whole run at the probe stage forever.

Summary

Types

What the probe decided for one mutant

What the probe decided for the whole run

Functions

Does selection hold a whole-suite run — :run_all, or any {:run, []} outcome (:full mode's covered mutants; an id covered only from an unlabeled process)? Mutare.Runner reads the umbrella dependency graph (a Mix boot) only when it does, since that graph narrows nothing else. Pure.

Build the per-mutant test selection (see selection/0).

Compute the per-mutant selection/0 from an already-decoded coverage dump.

Summarise a selection/0 for display (e.g. the --verbose coverage note): how many mutants got per-file / whole-suite selection (covered) versus were skipped as :no_coverage. :run_all (coverage unusable or uncertain) carries no per-mutant counts — every covered mutant runs the whole suite — so its counts are zero and run_all? is true. Pure (no IO), so it is unit-testable without a probe run.

Types

outcome()

@type outcome() :: {:run, [String.t()]} | :no_coverage

What the probe decided for one mutant:

  • {:run, test_args} — its line is covered; run mix test with these args ([] = whole suite; file-granular file paths otherwise; under :tests those file paths plus --only test:<name> flags narrowing to the covering tests).
  • :no_coverage — nothing runs its line; skip it and keep it out of the score's denominator.

selection()

@type selection() :: :run_all | {:selective, %{required(pos_integer()) => outcome()}}

What the probe decided for the whole run:

  • :run_all — coverage is unusable or uncertain (the probe failed, the dump couldn't be read, or capture tables were missing). Run every mutant against the whole suite — never skip on doubt.
  • {:selective, outcomes} — a per-mutant decision. outcomes is total: every mutant id maps to an explicit outcome, so a :no_coverage mutant is named, never implied by a missing key.

Functions

broad_runs?(arg1)

@spec broad_runs?(selection()) :: boolean()

Does selection hold a whole-suite run — :run_all, or any {:run, []} outcome (:full mode's covered mutants; an id covered only from an unlabeled process)? Mutare.Runner reads the umbrella dependency graph (a Mix boot) only when it does, since that graph narrows nothing else. Pure.

run(sandbox, schema, mode, opts \\ [])

@spec run(
  Path.t(),
  Mutare.Schema.t(),
  :tests | :coverage | :full,
  Mutare.Sandbox.Command.Invocation.run_opts()
) :: selection()

Build the per-mutant test selection (see selection/0).

Never fails: every uncertainty degrades to the conservative :run_all. The green check and timing live in Mutare.Runner.Baseline, which runs first.

opts are the probe run's options (Mutare.Sandbox.Command.Invocation.run_opts/0): a fixed :partition entry (e.g. MIX_TEST_PARTITION=1) when :partition_env is on, so the partitioned suite finds a valid database (the probe is a single sequential run, so one fixed partition suffices — Mutare.Runner.Partitions); the :max_heap_mb cap; and a :cap (ms, or nil for uncapped) bounding the probe's wall clock via the same injected self-halt watcher a per-mutant run uses — an overrun exits Mutare.Sandbox.Command.Exit.timeout/0 and degrades to :run_all like any other non-zero probe exit (see the moduledoc). The probe adds its own :coverage option; [] (the default) sets nothing else.

select(mode, schema, coverage)

@spec select(:tests | :coverage | :full, Mutare.Schema.t(), Mutare.Coverage.t()) ::
  {:selective, %{required(pos_integer()) => outcome()}}

Compute the per-mutant selection/0 from an already-decoded coverage dump.

The pure core of run/5 (no IO): given the mode, the Mutare.Schema (for the total id list), and a valid Mutare.Coverage.t(), it returns a total {:selective, outcomes}. An empty aggregate marks every mutant :no_coverage; run/5 handles probe failures and unusable dumps before selection. Exposed so the mode reconciliation — including :tests narrowing and its whole-file/whole-suite fallbacks — is unit-testable without spawning a probe.

summarize(arg1)

@spec summarize(selection()) :: %{
  covered: non_neg_integer(),
  no_coverage: non_neg_integer(),
  run_all?: boolean()
}

Summarise a selection/0 for display (e.g. the --verbose coverage note): how many mutants got per-file / whole-suite selection (covered) versus were skipped as :no_coverage. :run_all (coverage unusable or uncertain) carries no per-mutant counts — every covered mutant runs the whole suite — so its counts are zero and run_all? is true. Pure (no IO), so it is unit-testable without a probe run.