Mutare.Runner.CoverageProbe (mutare v0.1.2)

Copy Markdown View Source

Decide, per mutant, which test files to run — coverage-driven test selection.

Runs after Mutare.Runner.Baseline has confirmed the suite green and measured the timing; this module is purely about coverage and never green-checks or times anything. Keeping the two apart is the point: folding coverage into the baseline made baseline_ms a sum of per-file process boots (an inflated timeout cap), and meant the suite was never confirmed green together — only file-by-file.

It is one instrumented suite run. The probe runs mix test once at baseline with the coverage-capture flag set, so the metamutant self-records — in the test process, synchronously — which mutant ids each test file covers, plus a process-agnostic aggregate of every id that ran at all (Mutare.Coverage.Recorder owns the capture, Mutare.Coverage reads the dump). No :cover, no per-file subprocess fan-out, and no async-formatter race that loses fast async: false modules' coverage.

Three modes, set by :test_selection, a granularity ladder from safest/slowest to fastest — :full:coverage:tests:

  • :tests (default) — per mutant, run only the individual test cases that covered its line, via mix test <file> --only test:<name> (one --only per covering test). Finer than :coverage: where :coverage runs a whole covering file, :tests runs just the tests in it that touched the line. It falls back to :coverage's whole-file selection for any id whose coverage can't be pinned to runnable tests — an id in the wholefile set (covered through a setup_all/on_exit, which map to a module-scoped context, not a single test), or one covered via an unlabeled process (whole suite). So the narrowing is applied only where every covering attribution is a concrete, runnable test.
  • :coverage — per mutant, run only the test files that covered its line (a setup_all attributes to its own module's file via the __ex_unit__/2 stacktrace frame; a Task via its caller chain; a test-registered on_exit via its closure frame); a mutant whose code ran in an unlabeled process (a bare spawn, a setup-registered on_exit closure, or a setup_all whose work happened in a spawned Task) runs the whole suite — even if some file also attributes it, since that partial attribution would otherwise mask the unlabeled coverage and produce a false survivor; a mutant that never ran at all is :no_coverage (skipped, and kept out of the score's denominator).
  • :full — no per-file selection: every covered mutant runs the whole suite, the rest are :no_coverage. Safest for suites with cross-file dependencies — including a setup_all with cross-module global side effects, which :coverage attributes to its own file only (see Mutare.Coverage.Recorder).

:tests and the file-granular safety margin it trades away

:coverage is file-granular on purpose: if any test in a file covers the line, the whole file runs — so a sibling test that kills the mutant indirectly (fails because a covering test in the same async: false module ran the mutated line and left corrupt shared state, without the sibling touching the line itself) is still run. :tests narrows to the covering tests only, dropping that sibling — so on a stateful, cross-test-dependent suite :tests can turn such a kill into a false survivor. It is the default on the assumption that suites are predominantly async: true with per-test-isolated state (where a sibling that never runs the line cannot observe the mutation, so nothing is lost); :coverage is the opt-out (--per-file) and :full the fully conservative escape hatch.

The narrowing never manufactures a false kill: every name emitted is a test that actually ran in one of the included files during the probe, so at least one test always matches and mix test --only never hits its "no test executed" error (which would otherwise surface as a harness error, not a kill).

Coverage is advisory, never authoritative. Anything uncertain — a non-zero probe exit, an unreadable dump, or an empty dump (the probe recorded nothing, so the capture itself likely failed) — degrades to :run_all: we never skip a mutant on doubt. But because :run_all makes every covered mutant run the whole suite — prohibitive on a large project — a failed probe run is retried once before degrading: the baseline was green moments earlier, so a probe failure is usually a flaky test, and one extra suite run is cheap next to a whole run's selection quality. The probe run is wall-clock capped for the same reason (by default a generous multiple of the per-mutant cap, since instrumentation adds overhead a plain baseline doesn't have; :probe_timeout sets an explicit cap instead): a pathological interaction between the coverage capture and the target's hot loops must degrade to :run_all, not hang the whole run at the probe stage forever.

Summary

Types

What the probe decided for one mutant

What the probe decided for the whole run

Functions

Does selection hold a whole-suite run — :run_all, or any {:run, []} outcome (:full mode's covered mutants; an id covered only from an unlabeled process)? Mutare.Runner reads the umbrella dependency graph (a Mix boot) only when it does, since that graph narrows nothing else. Pure.

Build the per-mutant test selection (see selection/0).

Decide the per-mutant selection/0 from an already-decoded coverage dump.

Summarise a selection/0 for display (e.g. the --verbose coverage note): how many mutants got per-file / whole-suite selection (covered) versus were skipped as :no_coverage. :run_all (coverage unusable or uncertain) carries no per-mutant counts — every covered mutant runs the whole suite — so its counts are zero and run_all? is true. Pure (no IO), so it is unit-testable without a probe run.

Types

outcome()

@type outcome() :: {:run, [String.t()]} | :no_coverage

What the probe decided for one mutant:

  • {:run, test_args} — its line is covered; run mix test with these args ([] = whole suite; file-granular file paths otherwise; under :tests those file paths plus --only test:<name> flags narrowing to the covering tests).
  • :no_coverage — nothing runs its line; skip it and keep it out of the score's denominator.

selection()

@type selection() :: :run_all | {:selective, %{required(pos_integer()) => outcome()}}

What the probe decided for the whole run:

  • :run_all — coverage is unusable or uncertain (couldn't read the dump, or not a single id was recorded, which means the capture itself likely failed). Run every mutant against the whole suite — never skip on doubt.
  • {:selective, outcomes} — a per-mutant decision. outcomes is total: every mutant id maps to an explicit outcome, so a :no_coverage mutant is named, never implied by a missing key.

Functions

broad_runs?(arg1)

@spec broad_runs?(selection()) :: boolean()

Does selection hold a whole-suite run — :run_all, or any {:run, []} outcome (:full mode's covered mutants; an id covered only from an unlabeled process)? Mutare.Runner reads the umbrella dependency graph (a Mix boot) only when it does, since that graph narrows nothing else. Pure.

run(sandbox, schema, mode, env \\ [], cap \\ nil)

@spec run(
  Path.t(),
  Mutare.Schema.t(),
  :tests | :coverage | :full,
  [{String.t(), String.t()}],
  pos_integer() | nil
) :: selection()

Build the per-mutant test selection (see selection/0).

Never fails: every uncertainty degrades to the conservative :run_all. The green check and timing live in Mutare.Runner.Baseline, which runs first.

env is extra environment for the probe run — a fixed partition entry (e.g. MIX_TEST_PARTITION=1) when :partition_env is on, so the partitioned suite finds a valid database; [] (the default) adds none. The probe is a single sequential run, so one fixed partition suffices (Mutare.Runner.Partitions).

cap (ms, or nil for uncapped) bounds the probe's wall clock via the same injected self-halt watcher a per-mutant run uses; an overrun exits Mutare.Sandbox.Command.timeout_exit/0 and degrades to :run_all like any other non-zero probe exit (see the moduledoc).

select(mode, schema, coverage)

@spec select(:tests | :coverage | :full, Mutare.Schema.t(), Mutare.Coverage.t()) ::
  selection()

Decide the per-mutant selection/0 from an already-decoded coverage dump.

The pure core of run/5 (no IO): given the mode, the Mutare.Schema (for the total id list), and a Mutare.Coverage.t(), it returns :run_all (empty aggregate — the capture recorded nothing, so it likely failed) or a total {:selective, outcomes}. Exposed so the mode reconciliation — including :tests narrowing and its whole-file/whole-suite fallbacks — is unit-testable without spawning a probe.

summarize(arg1)

@spec summarize(selection()) :: %{
  covered: non_neg_integer(),
  no_coverage: non_neg_integer(),
  run_all?: boolean()
}

Summarise a selection/0 for display (e.g. the --verbose coverage note): how many mutants got per-file / whole-suite selection (covered) versus were skipped as :no_coverage. :run_all (coverage unusable or uncertain) carries no per-mutant counts — every covered mutant runs the whole suite — so its counts are zero and run_all? is true. Pure (no IO), so it is unit-testable without a probe run.