Mutare.Runner.CoverageProbe (mutare v0.4.1)

Copy Markdown View Source

Select the test files to run for each mutant — coverage-driven test selection.

Runs after Mutare.Runner.Baseline has confirmed the suite green and measured the timing; this module is purely about coverage and never green-checks or times anything. Keeping the two apart is the point: folding coverage into the baseline made baseline_ms a sum of per-file process boots (an inflated timeout cap), and meant the suite was never confirmed green together — only file-by-file.

It is one instrumented suite run. The probe runs mix test once at baseline with the coverage-capture flag set, so the metamutant self-records — in the test process, synchronously — which mutant ids each test file covers, plus a process-agnostic aggregate of every id that ran at all (Mutare.Coverage.Recorder implements the capture, Mutare.Coverage reads the dump). No :cover, no per-file subprocess fan-out, and no async-formatter race that loses fast async: false modules' coverage.

Three modes, set by :test_selection, a granularity ladder from safest/slowest to fastest — :full ⊃ :coverage ⊃ :tests:

  • :tests (default) — per mutant, run only the individual test cases that covered its line, via mix test <file> --only test:<name> (one --only per covering test). Finer than :coverage: where :coverage runs a whole covering file, :tests runs just the tests in it that touched the line. It falls back to :coverage's whole-file selection for any id whose coverage can't be pinned to runnable tests — an id in the wholefile set (covered through a setup_all/on_exit, which map to a module-scoped context, not a single test), or one covered via an unlabeled process (whole suite). So the narrowing is applied only where every covering attribution is a concrete, runnable test.
  • :coverage — per mutant, run only the test files that covered its line (a setup_all attributes to its own module's file via the __ex_unit__/2 stacktrace frame; a Task via its caller chain; a test-registered on_exit via its closure frame); a mutant whose code ran in an unlabeled process (a bare spawn, a setup-registered on_exit closure, or a setup_all whose work happened in a spawned Task) runs the whole suite — even if some file also attributes it, since that partial attribution would otherwise mask the unlabeled coverage and produce a false survivor; a mutant that never ran at all is :no_coverage (skipped, and kept out of the score's denominator).
  • :full — no per-file selection: every covered mutant runs the whole suite, the rest are :no_coverage. Safest for suites with cross-file dependencies — including a setup_all with cross-module global side effects, which :coverage attributes to its own file only (see Mutare.Coverage.Recorder).

:tests and the file-granular safety margin it trades away

:coverage is file-granular on purpose: if any test in a file covers the line, the whole file runs — so a sibling test that kills the mutant indirectly (fails because a covering test in the same async: false module ran the mutated line and left corrupt shared state, without the sibling touching the line itself) is still run. :tests narrows to the covering tests only, dropping that sibling — so on a stateful, cross-test-dependent suite :tests can turn such a kill into a false survivor. It is the default on the assumption that suites are predominantly async: true with per-test-isolated state (where a sibling that never runs the line cannot observe the mutation, so nothing is lost); :coverage is the opt-out (--per-file) and :full the fully conservative escape hatch.

The narrowing never manufactures a false kill: every name emitted is a test that actually ran in one of the included files during the probe, so at least one test always matches and mix test --only never hits its "no test executed" error (which would otherwise surface as a harness error, not a kill).

Coverage is advisory, never authoritative. Anything uncertain — a non-zero probe exit, an unreadable dump, or missing capture tables — degrades to run-all ({:run_all, degrade}, the reason attached): we never skip a mutant on doubt. A valid empty dump means none of the emitted mutants ran; focused selection can legitimately leave the entire aggregate empty. But because run-all makes every covered mutant run the whole suite — prohibitive on a large project — a failed probe run is retried once before degrading: the baseline was green moments earlier, so a probe failure is usually a flaky test, and one extra suite run is cheap next to a whole run's selection quality. The probe run is wall-clock capped for the same reason (by default a generous multiple of the per-mutant cap, since instrumentation adds overhead a plain baseline doesn't have; :probe_timeout sets an explicit cap instead): a pathological interaction between the coverage capture and the target's hot loops must degrade to run-all, not hang the whole run at the probe stage forever.

Every path into "run the whole suite" is loud. The selection's summarize/1 counts whole-suite runs apart from narrowed ones and carries the run-all reason, and broad_ids/1 names the mutants concerned, so Mutare.Runner's {:coverage_done, …} event lets Mutare.Report.Live say so in every mode but --quiet and mark each such mutant as it runs — a silent whole-suite run reads as a hang, or as Mutare being slow.

Summary

Types

Why the probe degraded to run-all (selection/0): the probe run exited non-zero (:probe_failed, with its exit status), it overran its wall-clock cap (:probe_timed_out, with the cap in ms), or it exited green but its dump was unreadable (:dump_unreadable, with Mutare.Coverage.read_dump/2's error). Data, not text: Mutare.Report.Live.Lines renders it, and the API caller reads it.

What the probe decided for one mutant

What the probe decided for the whole run

The shape of one mutant's test run, read off its outcome/0 args by shape/1: :suite (the whole suite), :files (its covering test files), or :tests (those files narrowed to the covering test cases). Mutare.Result.selection records the shape a mutant actually ran, with :app for a whole-suite run an umbrella narrowed to the owning app and its dependents (Mutare.Runner.MutantRun).

Functions

The ids whose run is the whole suite ({:run, []}) in a selective selection — what the live display marks in flight, so a run that takes fifty times its neighbours is explained on the line that shows it. :all under run-all. Pure.

Does selection hold a whole-suite run — run-all, or any {:run, []} outcome (:full mode's covered mutants; an id covered only from an unlabeled process)? Mutare.Runner reads the umbrella dependency graph (a Mix boot) only when it does, since that graph narrows nothing else. Pure.

Build the per-mutant test selection (see selection/0).

Compute the per-mutant selection/0 from an already-decoded coverage dump.

The shape/0 of a run with these mix test args: [] is the whole suite, args carrying a --only filter are narrowed test cases, anything else is whole files. The reader of the argv select/3 builds (only_args/1 writes the flag this reads).

Summarise a selection/0 for display (the {:coverage_done, …} phase event Mutare.Report.Live renders): how many mutants run under each shape/0 — tests, files, suite — versus were skipped as no_coverage. A whole-suite run under :tests/:coverage is the slowdown nobody asked for, so it is counted apart, never folded into a "covered" total. {:run_all, degrade} (coverage unusable or uncertain) carries no per-mutant counts — every covered mutant runs the whole suite — so its counts are zero, run_all? is true, and degrade says why. Pure (no IO), so it is unit-testable without a probe run.

Types

degrade()

@type degrade() ::
  %{cause: :probe_failed, exit_status: non_neg_integer()}
  | %{cause: :probe_timed_out, cap_ms: pos_integer() | nil}
  | %{cause: :dump_unreadable, error: term()}

Why the probe degraded to run-all (selection/0): the probe run exited non-zero (:probe_failed, with its exit status), it overran its wall-clock cap (:probe_timed_out, with the cap in ms), or it exited green but its dump was unreadable (:dump_unreadable, with Mutare.Coverage.read_dump/2's error). Data, not text: Mutare.Report.Live.Lines renders it, and the API caller reads it.

outcome()

@type outcome() :: {:run, [String.t()]} | :no_coverage

What the probe decided for one mutant:

  • {:run, test_args} — its line is covered; run mix test with these args ([] = whole suite; file-granular file paths otherwise; under :tests those file paths plus --only test:<name> flags narrowing to the covering tests).
  • :no_coverage — nothing runs its line; skip it and keep it out of the score's denominator.

selection()

@type selection() ::
  {:run_all, degrade()} | {:selective, %{required(pos_integer()) => outcome()}}

What the probe decided for the whole run:

  • {:run_all, degrade} — coverage is unusable or uncertain (the probe failed, the dump couldn't be read, or capture tables were missing — degrade/0 says which). Run every mutant against the whole suite — never skip on doubt.
  • {:selective, outcomes} — a per-mutant decision. outcomes is total: every mutant id maps to an explicit outcome, so a :no_coverage mutant is named, never implied by a missing key.

shape()

@type shape() :: :suite | :files | :tests

The shape of one mutant's test run, read off its outcome/0 args by shape/1: :suite (the whole suite), :files (its covering test files), or :tests (those files narrowed to the covering test cases). Mutare.Result.selection records the shape a mutant actually ran, with :app for a whole-suite run an umbrella narrowed to the owning app and its dependents (Mutare.Runner.MutantRun).

Functions

broad_ids(arg)

@spec broad_ids(selection()) :: :all | MapSet.t(pos_integer())

The ids whose run is the whole suite ({:run, []}) in a selective selection — what the live display marks in flight, so a run that takes fifty times its neighbours is explained on the line that shows it. :all under run-all. Pure.

broad_runs?(arg)

@spec broad_runs?(selection()) :: boolean()

Does selection hold a whole-suite run — run-all, or any {:run, []} outcome (:full mode's covered mutants; an id covered only from an unlabeled process)? Mutare.Runner reads the umbrella dependency graph (a Mix boot) only when it does, since that graph narrows nothing else. Pure.

run(sandbox, schema, mode, opts \\ [])

@spec run(
  Path.t(),
  Mutare.Schema.t(),
  :tests | :coverage | :full,
  Mutare.Sandbox.Command.Invocation.run_opts()
) :: selection()

Build the per-mutant test selection (see selection/0).

Never fails: every uncertainty degrades to the conservative run-all. The green check and timing live in Mutare.Runner.Baseline, which runs first.

opts are the probe run's options (Mutare.Sandbox.Command.Invocation.run_opts/0): a fixed :partition entry (e.g. MIX_TEST_PARTITION=1) when :partition_env is on, so the partitioned suite finds a valid database (the probe is a single sequential run, so one fixed partition suffices — Mutare.Runner.Partitions); the :max_heap_mb cap; the mutants' :schedulers trim, so the probe executes what a mutant run will (a suite may branch on the scheduler count, and ExUnit's concurrency follows it); and a :cap (ms, or nil for uncapped) bounding the probe's wall clock via the same injected self-halt watcher a per-mutant run uses — an overrun exits Mutare.Sandbox.Command.Exit.timeout/0 and degrades to run-all like any other non-zero probe exit (see the moduledoc). The probe adds its own :coverage option; [] (the default) sets nothing else.

select(mode, schema, coverage)

@spec select(:tests | :coverage | :full, Mutare.Schema.t(), Mutare.Coverage.t()) ::
  {:selective, %{required(pos_integer()) => outcome()}}

Compute the per-mutant selection/0 from an already-decoded coverage dump.

The pure core of run/4 (no IO): given the mode, the Mutare.Schema (for the total id list), and a valid Mutare.Coverage.t(), it returns a total {:selective, outcomes}. An empty aggregate marks every mutant :no_coverage; run/4 handles probe failures and unusable dumps before selection. Exposed so the mode reconciliation — including :tests narrowing and its whole-file/whole-suite fallbacks — is unit-testable without spawning a probe.

shape(args)

@spec shape([String.t()]) :: shape()

The shape/0 of a run with these mix test args: [] is the whole suite, args carrying a --only filter are narrowed test cases, anything else is whole files. The reader of the argv select/3 builds (only_args/1 writes the flag this reads).

summarize(arg)

@spec summarize(selection()) :: %{
  tests: non_neg_integer(),
  files: non_neg_integer(),
  suite: non_neg_integer(),
  no_coverage: non_neg_integer(),
  run_all?: boolean(),
  degrade: degrade() | nil
}

Summarise a selection/0 for display (the {:coverage_done, …} phase event Mutare.Report.Live renders): how many mutants run under each shape/0 — tests, files, suite — versus were skipped as no_coverage. A whole-suite run under :tests/:coverage is the slowdown nobody asked for, so it is counted apart, never folded into a "covered" total. {:run_all, degrade} (coverage unusable or uncertain) carries no per-mutant counts — every covered mutant runs the whole suite — so its counts are zero, run_all? is true, and degrade says why. Pure (no IO), so it is unit-testable without a probe run.