Mutare.Runner.Baseline (mutare v0.1.0)

Copy Markdown View Source

Run the complete test suite green-checked against the baseline mutant (MUTARE_ACTIVE_MUTANT=0) — once, several times to catch a flaky suite, or with retries to survive an occasionally red startup run.

This is the authoritative green check and the source of baseline_ms. Two things hang on it being a run of the whole suite:

  • Timing. baseline_ms is the wall-clock of one process boot plus the suite, which the runner scales into the per-mutant timeout cap. Summing per-file probe runs (as the old conflated probe did) folded N process boots into the figure and inflated every mutant's cap. When the suite runs more than once (see flakiness, below) we take the slowest green run, so the cap stays conservative.
  • Green-ness. The suite is confirmed green together. A per-file probe never runs the suite as a whole, so a cross-file dependency could pass file-by-file (or fail in isolation) without the suite's real state ever being checked.

Mutation testing on a red suite is meaningless — every "kill" is suspect — so a non-green baseline aborts the run: {:error, :baseline_failed, output}. When :baseline_retries is non-zero, an all-red baseline attempt is retried before aborting; a mixed pass/fail attempt still aborts as flaky immediately. No --cover here: mutant runs don't use it, so an uninstrumented baseline times the cap against like conditions (and the --cover instrumentation belongs to Mutare.Runner.CoverageProbe, which runs separately afterwards).

Flakiness (:baseline_runs)

A flaky test — one that passes/fails nondeterministically regardless of the mutant — is corrosive here: when it goes red during a mutant's run it marks that mutant killed, a false kill that hides a real survivor. Flakiness is a property of the suite, not of any one mutant, so the cheapest place to catch it is right here: run the baseline up to :baseline_runs times (default 1) and classify the outcomes (classify/1):

  • all green → proceed ({:ok, slowest_green_ms});
  • all red → {:error, :baseline_failed, output} (a deterministically broken suite — unchanged from a single run);
  • mixed → a test disagreed with itself → {:error, :baseline_flaky, detail}, aborting loudly with the flaky tests named, rather than scoring against a suite that manufactures false kills.

Collection short-circuits the moment outcomes disagree — flakiness is already proven, so there's no point running the rest. With :baseline_runs at its default of 1 this is exactly the old single-run behavior: one run, green or red.

Retry survival (:baseline_retries)

:baseline_retries (default 0) wraps the whole :baseline_runs check: if an attempt is consistently red (:baseline_failed), Mutare tries the baseline again up to that many times. This is for startup/load/order flakes where the project may be green on a later run. It deliberately does not retry :baseline_flaky: when one :baseline_runs attempt observes both pass and fail, the suite has already disagreed with itself and scoring against it would hide false kills.

Summary

Types

outcome()
result()

Functions

classify(outcomes)

Classify a non-empty list of baseline run outcomes into a run result.

run(sandbox, runs \\ 1, env \\ [])

Run the whole suite up to runs times at baseline, then classify (classify/1). Returns {:ok, baseline_ms} when consistently green, {:error, :baseline_failed, output} when consistently red, or {:error, :baseline_flaky, detail} when the runs disagree.

run(sandbox, runs, retries, env)

Run the baseline check with retries extra all-red attempts before returning :baseline_failed.

Types

outcome()

@type outcome() :: {:pass, non_neg_integer()} | {:fail, String.t()}

result()

@type result() ::
  {:ok, non_neg_integer()}
  | {:error, :baseline_failed, String.t()}
  | {:error, :baseline_flaky, String.t()}

Functions

classify(outcomes)

@spec classify([outcome()]) :: result()

Classify a non-empty list of baseline run outcomes into a run result.

Pure: the I/O lives in run/2, the decision (and the flaky message) here, so it is unit-testable without spawning mix. All green → {:ok, slowest_ms}; all red → {:error, :baseline_failed, _}; a mix of both → {:error, :baseline_flaky, _}.

run(sandbox, runs \\ 1, env \\ [])

@spec run(Path.t(), pos_integer(), [{String.t(), String.t()}]) :: result()

Run the whole suite up to runs times at baseline, then classify (classify/1). Returns {:ok, baseline_ms} when consistently green, {:error, :baseline_failed, output} when consistently red, or {:error, :baseline_flaky, detail} when the runs disagree.

env is extra environment for each run — a fixed partition entry (e.g. MIX_TEST_PARTITION=1) when :partition_env is on, so a partitioned suite finds a valid database; [] (the default) adds none. The baseline is sequential, so one fixed partition suffices (Mutare.Runner.Partitions).

run(sandbox, runs, retries, env)

@spec run(Path.t(), pos_integer(), non_neg_integer(), [{String.t(), String.t()}]) ::
  result()

Run the baseline check with retries extra all-red attempts before returning :baseline_failed.

A retry wraps the full runs sample. :baseline_flaky is never retried because a mixed pass/fail sample has already proven suite nondeterminism.