Compile once, then run the suite once per mutant in a fresh OS process.

The flow protects the one-compile invariant: we compile the sandbox a single time, run the tests as a baseline to ensure it passes, then launch one mix test process per mutant with MUTARE_ACTIVE_MUTANT set. Sources never change between runs, so mix's incremental compiler finds nothing to rebuild — the per-mutant cost is process boot plus the suite (only up to the first failure for a kill), never recompilation.

The compile step distinguishes Mix dependency validation from actual compile-poisoning. A dependency failure returns :dependency_failed immediately: dropping mutant ids cannot repair copied dependency state, so it never enters poison recovery. The Mix task then points remediation at the original project rather than the disposable sandbox. The compile also carries a wall-clock cap (:compile_timeout, default 30 minutes, nil to disable): a config-hosted sibling of the per-mutant timeout watcher self-halts a pathological compile, surfaced as :compile_timed_out — likewise never fed to poison recovery, since there is no error to attribute and a rebuild cannot make an oversized compile faster.

run/2 returns {:ok, %Mutare.Run{}}: the Mutare.Schema that was run, the list of per-mutant Mutare.Results, the sandbox path, the baseline run's wall-clock in milliseconds, and whether an early-stop condition (:max_survivors or :time_budget) stopped the run early.

Baseline + coverage probe

Before the per-mutant loop we run the test suite once and ensure it passes (Mutare.Runner.Baseline) — then build a per-mutant test selection (Mutare.Runner.CoverageProbe) which picks the test files each mutant needs (or marks it :no_coverage). The two are split on purpose: a failing baseline test aborts, while coverage is advisory and degrades to running everything. The baseline can be run more than once (:baseline_runs) to catch a flaky suite: runs that disagree abort with :baseline_flaky rather than let a flaky test manufacture false mutant kills. A consistently red baseline attempt can also be retried (:baseline_retries, default 0) to survive startup/load flakes without weakening :baseline_runs' mixed-outcome check. See those modules for the selection modes.

Unanimous kill reruns

:kill_runs (--kill-runs, default 1) handles the narrower case where a residual flaky test appears only under one mutant's timing. Only kill outcomes are rerun, and every attempt must kill. If a later attempt passes, the mutant is recorded as :survived; if a later attempt persistently hits the harness, it remains :harness_error. Harness retries are the inner infrastructure layer; kill reruns combine settled test-suite verdicts.

Parallel workers and timeouts

The per-mutant phase runs :workers mutants concurrently (default: half System.schedulers_online/0, capped at 4 — each worker is a full mix test BEAM that itself uses every scheduler, so a parallel suite already scales with the machine and extra workers only fill the serial/IO gaps one run leaves), each its own OS process in the shared sandbox. Each run has a wall-clock cap: an explicit :timeout in ms, or baseline × :timeout_multiplier (default 3.0) scaled by half the concurrent lanes — the baseline is measured uncontended, so wall time under contention legitimately inflates with the lane count — with a floor. A mutation can turn a terminating loop infinite, so the run is capped; a capped run counts as :timeout — a kill, since the hang is observable misbehavior.

Even a scaled cap can be overrun by a slow-but-finite run, and survivors are the most exposed (a kill exits at its first failing test; a survivor must run its entire selected set). A false :timeout is a false kill hiding a true survivor, so by default (:confirm_timeouts) a streamed :timeout is provisional: after the stream drains, each timed-out mutant is re-run sequentially — no contention — with the same cap, and that verdict is recorded instead. Only a repeat overrun records :timeout; a genuine hang pays one extra cap. confirm_timeouts: false (--no-confirm-timeouts) records the first overrun as-is. A wall-clock :time_budget covers this confirmation pass too: confirmations already launched are allowed to finish, but no new confirmation run starts after the deadline.

Early stop: survivor cap (:max_survivors) or time budget (:time_budget)

Two conditions can stop the per-mutant loop before every mutant runs, whichever fires first. :max_survivors (--max-survivors) stops once that many survivors (:survived results) have surfaced — an iterate-and-fix workflow that wants a handful of concrete test gaps rather than a full run. :time_budget (--time-budget, a duration string like "10m" parsed by Mutare.Duration) stops once that much wall-clock elapses in the per-mutant phase — a "see what I can get in ten minutes" run. The clock starts as the phase begins (compile/baseline/probe are not charged against it) and is checked just before a task announces and launches a real mutant run, so ordered result buffering cannot hide an expired budget and allow more mutants to start.

Unlike :max_mutants (a Mutare.Schema cap on candidate sites), both leave every mutant compiled in — only the run halts early. The per-mutant stream is consumed ordered: true, so a survivor stop is deterministic: the Nth survivor in source order, regardless of which worker finished first, and the reported survivors are exactly the first N. (A time-budget stop is not deterministic — it depends on how far the run got.) Runs already in flight when either condition trips are drained (not killed), so the sandbox teardown never races a live mix subprocess. If the budget elapses after every mutant has already launched, the result set is still complete unless the budget also prevents a provisional timeout from being confirmed. Otherwise the returned run carries stopped_early; on an early stop the harness-error abort guard is skipped (the score is already budget-limited or a partial prefix — the Mix task notes it and skips the --min-score gate too), since aborting would discard the very survivors the user asked to find.

Per-worker partitioning (DB isolation)

Optionally (:partition_env, off by default), each concurrent run is handed a distinct partition id under a named env var (default MIX_TEST_PARTITION), so a stateful suite can point each worker at its own database — the mix test --partitions convention. The ids come from a bounded, recycled pool (Mutare.Runner.Partitions) sized to :workers, so two live runs never share a partition and only :workers databases are needed. The one compile, the baseline, and the coverage probe (all sequential, pre-pool) take a fixed partition — the compile too, since it evaluates the target's config, where a partitioned default-less System.fetch_env! would otherwise raise. Inert when unset.

Harness errors are kept out of the score

A mutant run that never reaches a verdict — a compile error, a missing dependency, a filesystem race — says nothing about the mutation, so it is recorded as :harness_error and kept out of the score's denominator, never silently miscounted as a kill the way a raw "non-zero ⇒ killed" rule would.

Two knobs harden this against flakiness and systemic breakage:

• :harness_retries (default 2) re-runs a harness-errored mutant before recording it, so a transient failure (a filesystem/lock race) gets another chance; a real verdict is never retried.
• :max_harness_error_rate (default 0.5, nil to disable) aborts the whole run — {:error, :too_many_harness_errors, detail} — when persistent harness errors exceed that fraction of the mutants that ran. Past that, the sandbox is broken, not the mutations tested, and a score over the surviving denominator would mislead; better to fail loudly.

Boot-failure: a known-transient harness error retried harder

One harness-error cause is recognised by name (Output.boot_failure?/1 → the :boot_failure outcome): the sandbox node dies during boot with its own diagnostic erased by a secondary :standard_error failure. It is almost always concurrent workers contending on shared singletons at startup (a test DB, a connection pool), so it clears on a retry that doesn't re-collide with the boot stampede. It gets its own retry budget (@boot_failure_retries), independent of :harness_retries and with a short jittered backoff, plus a specific warning that stops pointing at output that can't help (the real cause is unrecoverable) and names the actual contention levers — --workers and --partition-db/--partition-env. (Not --harness-retries: a :boot_failure draws only from its own dedicated budget, so raising that knob would not retry it more.) The verdict is unchanged (a harness error, out of the score); only the messaging and retry effort differ.

SIGKILL (likely OOM): a harness error never retried, and the :max_heap_mb cap

The mirror-image refinement: a run the OS killed with SIGKILL (exit 137, the :sigkilled outcome) is recognised so it is never retried — the opposite of :boot_failure. Its signature cause is the kernel OOM killer reaping a mutant whose mutation made it allocate without bound (a dropped guard turning a function unconditionally self-recursive can exhaust tens of GB in under a second — faster than any wall-clock watcher can react), and that failure is deterministic: a back-to-back retry re-detonates the same blowup on the host. The verdict stays a harness error (out of the score), with a specific warning naming the likely cause and the mitigation.

The mitigation is :max_heap_mb (--max-heap-mb, off by default): a per-process BEAM heap cap injected into every runtime sandbox run — baseline, coverage probe, per-mutant — so a runaway-allocation mutant dies as an ordinary, fast test failure inside its own run instead of endangering the host. The baseline running under the same cap validates up front that the suite itself fits under it. The one metamutant compile is deliberately not capped. Mechanism and sizing guidance: Mutare.Sandbox.Command.Invocation.heap_cap_env/1.