Mutare.Sandbox.Command (mutare v0.3.1)

Decode what a mix test mutant run did, and orchestrate one typed run.

Every mutant is exercised by its own mix test process (spawned by Mutare.Sandbox.Command.Invocation). This module implements the decoding side of the exit-code contract — reading a run's exit code and output into a mutant verdict — plus the one entry point that runs a mutant and hands back a typed Mutare.Sandbox.Command.Result. The neighbouring modules implement the rest of what was once one file:

Mutare.Sandbox.Command.Exit — the exit codes themselves and their by-code reading (Exit.decode/1), the leaf both sides of the contract read from.
Mutare.Sandbox.Command.Invocation — process execution, the environment a run gets (one builder per run option, from which the reserved variable set derives), and the watcher ASTs (the producing side of the timeout and owner-death codes).
Mutare.Sandbox.Command.Output — every pattern that reads mix's human-readable output, including the discriminators used by outcome/2 below.
Mutare.Sandbox.CompilerOptions — the env that speeds the one metamutant compile.

The exit-code contract

The codes themselves — and the total by-code reading, decode/1 — live in the leaf Mutare.Sandbox.Command.Exit, which both the watcher ASTs (the producing side, in Invocation) and this module (the decoding side) read from.

outcome/2 refines Exit.decode/1's ambiguous "anything else" case with the run's output (via the Mutare.Sandbox.Command.Output discriminators). Three refinements recover detected-mutant cases from the otherwise-:harness_error bucket:

a mutation that broke the test suite's own compilation (it ran at the test modules' compile time) exits 1 with a test-script compile-error banner (Output.suite_compile_error?/1) — a kill, not infra.
a mutation that minted unbounded atoms (an unterminated search building a fresh :"#{x}_#{i}" per step) crashes the BEAM when the global atom table fills (Output.atom_exhausted?/1) — a resource-divergence exactly like a CPU-bound timeout (the suite can never pass with it), so also a kill. The VM aborts before the in-process timeout watcher can self-halt, which is why it surfaces here rather than as a clean Exit.timeout/0.
a mutation reachable from application startup stops mix test before it loads a single test: the sandbox selects the mutant in the mix.exs prefix, project evaluation fails (Output.project_failure?/1), runtime configuration raises (Output.config_failure?/1), or app.start raises and Mix exits 1 with Could not start application … (Output.app_start_failure?/1). The baseline boots the same sandbox green, so the mutation is what broke it — a kill, on the same reasoning as the suite-compile case. Mutare.Runner.MutantRun spends the boot-contention retry budget first, so only a persistent one is charged.

A fourth refinement does not change the verdict — it stays a harness error — but names a known-transient cause so the runner can message and retry it better (Output.boot_failure?/1 → :boot_failure): the sandbox node died during boot and its own diagnostic was erased by a secondary :standard_error failure (a torn-down IO device). This provides no verdict on the mutation: concurrent workers are contending on shared singletons at startup — so it is kept out of the score like any harness error, but it is recognised here so the engine stops pointing at output that can't help (the real cause is unrecoverable) and retries it harder (see Mutare.Runner).

timed_test/4 applies the --exit-status flag (test_argv/1), runs the mutant via Mutare.Sandbox.Command.Invocation.timed_mix/4, and returns a typed Mutare.Sandbox.Command.Result decoded via outcome/2.

Kill detection stops at the first failure

A mutant is killed the moment any test fails — the verdict is killed-vs-survived, not which tests fail — so timed_test/4 also forces --max-failures 1. ExUnit then stops scheduling tests at the first failure, which is a strict speedup on the kill path (the common case for a healthy suite) and changes nothing else: the exit-status path is driven solely by failures > 0 (so one failure still exits Exit.failure/0 → :failed), and an all-pass survivor run never reaches the cap, so it still runs the whole (selected) suite to confirm survival. This is the kill path only — the baseline (Mutare.Runner.Baseline, a whole-suite green check and the timing source) and the coverage probe (Mutare.Runner.CoverageProbe, which must run every test to capture coverage) bypass timed_test/4 and are unaffected.

Summary

Types

outcome()

What a mix test mutant run did, decoded from its exit status (and, for the last case, its output)

Functions

outcome(status, output)

Decode an exit status refined by the run's output — the only place this contract looks past the exit code, and only to split one ambiguous case.

test_argv(test_args)

Build the mix test argv for a mutant kill-detection run from test_args.

timed_test(sandbox, test_args, mutant_id, opts \\ [])

Run the suite against mutant mutant_id, wall-clock-timed, and return a typed Mutare.Sandbox.Command.Result.

Types

outcome()

@type outcome() ::
  :passed
  | :failed
  | :timeout
  | :harness_error
  | :suite_compile_error
  | :atom_exhausted
  | :app_start_failure
  | :boot_failure
  | :sigkilled

What a mix test mutant run did, decoded from its exit status (and, for the last case, its output):

:passed — exit 0: every test passed despite the mutation.
:failed — exit Exit.failure/0: a test failed (a clean ExUnit failure).
:timeout — exit Exit.timeout/0: the watcher self-halted an overrun.
:harness_error — any other exit: the suite never ran to a verdict (a compile error, a missing dependency, a filesystem race, an OS signal). Not a statement about the mutation — the harness itself failed.
:suite_compile_error — a refinement of :harness_error: the test suite failed to compile because the mutation broke code that runs at the test modules' compile time (a Plug.Router route macro calling a mutated Plug.Router.Utils helper, an EEx/use-time call, a compile-time @attr expression…). The mutation was detected — the suite can't even build with it — so the runner counts it as a kill, not an infra failure (see outcome/2).
:atom_exhausted — a refinement of :harness_error: the mutation made the program mint unbounded atoms and the BEAM aborted when the atom table filled. A resource-divergence like a timeout (the suite can never pass with it), so the runner counts it as a kill — see outcome/2 and Output.atom_exhausted?/1.
:app_start_failure — a refinement of :harness_error: Mix refused to start the target's OTP application because the mutation broke project evaluation (Output.project_failure?/1), configuration (Output.config_failure?/1) or code reachable from Application.start/2 (Output.app_start_failure?/1). The suite never ran, but the mutation was detected — the app can't even boot with it — so the runner counts it as a kill once the boot-contention retries are spent (see outcome/2 and Mutare.Runner.MutantRun).
:boot_failure — a refinement of :harness_error that is still not a kill: the sandbox node died during boot and its own diagnostic was erased by a secondary :standard_error failure (Output.boot_failure?/1). A known- transient contention signature (concurrent workers stampeding shared services at startup), kept out of the score like any harness error but named so the runner messages it actionably and retries it harder.
:sigkilled — exit Exit.sigkill/0: the OS SIGKILLed the run. A harness error by verdict, but recognised so the runner never retries it: the dominant cause is the kernel OOM killer reaping a mutant made to allocate unboundedly, and re-running such a mutant re-detonates the same memory blowup on the host (see Mutare.Runner and the :max_heap_mb option).

Functions

outcome(status, output)

@spec outcome(non_neg_integer(), String.t()) :: outcome()

Decode an exit status refined by the run's output — the only place this contract looks past the exit code, and only to split one ambiguous case.

Exit 1 covers both a real harness failure (a missing dep, an infra compile error) and a mutation that broke the test suite's own compilation. They are indistinguishable by code, but they are not the same verdict: the latter means the mutation was detected (the suite can't build with it), so it is a kill, not an infra failure left out of the score.

We can tell them apart because the metamutant lib is compiled once before any mutant runs (poison handled at baseline), so a fresh compilation error during a per-mutant mix test can't come from the lib — it can only be a re-evaluated .exs test file the mutation broke at load time. So when an otherwise-:harness_error run's output reports a compilation error in a test script (Output.suite_compile_error?/1), it is :suite_compile_error. A second refinement recovers :atom_exhausted — a VM abort from the mutation minting unbounded atoms (Output.atom_exhausted?/1), a detected resource-divergence. Both are kills. So is a third — :app_start_failure (Output.app_start_failure?/1) — where Mix refused to start the application because the mutation broke project evaluation (Output.project_failure?/1), configuration (Output.config_failure?/1), or code reachable from Application.start/2; the baseline boots the same sandbox green, so the mutation is what stopped it. A fourth — :boot_failure (Output.boot_failure?/1) — stays a harness error but names a known-transient boot-time contention crash, so the runner can message and retry it better. Everything else (a lib-file compile error, a missing dep, no marker at all) stays :harness_error — fail safe: an ambiguous failure is never a kill.

The two boot markers are ordered :boot_failure first, deliberately. A node that died mid-boot with its own diagnostic erased is unambiguously infrastructure, whatever Mix printed on the way down; a Could not start application banner on its own is not. Startup contention that does reach Mix therefore still lands on :app_start_failure — which is why Mutare.Runner.MutantRun spends the dedicated boot-contention retry budget on it before recording the kill, rather than charging the first attempt.

:sigkilled (exit Exit.sigkill/0) deliberately bypasses the output refinements: a SIGKILLed run's output is truncated wherever the kill landed, so matching output markers in it would be unreliable — and none of these markers' causes exits via SIGKILL anyway (a compile error exits 1; atom exhaustion is the VM aborting itself).

test_argv(test_args)

@spec test_argv([String.t()]) :: [String.t()]

Build the mix test argv for a mutant kill-detection run from test_args.

Always prepends test --exit-status 101 --max-failures 1 plus the boot-skip flags --no-compile --no-deps-check --no-archives-check:

--exit-status 101 makes a clean test failure (a kill) distinguishable from a harness error — see the moduledoc.
--max-failures 1 stops ExUnit at the first failure, since one failing test is enough to declare a kill (also see the moduledoc).
--no-compile --no-deps-check --no-archives-check skip mix startup checks that are pure overhead under the one-compile invariant (the lib is built once, sources never change between runs) — see @boot_skip_flags.

test_args are the extra arguments ([] = whole suite, file-granular args otherwise), appended last so file-granular selection stays at the tail. Pure, so the contract is unit-testable without spawning mix.

timed_test(sandbox, test_args, mutant_id, opts \\ [])

@spec timed_test(
  Path.t(),
  [String.t()],
  Mutare.RuntimeId.t(),
  Mutare.Sandbox.Command.Invocation.run_opts()
) :: Mutare.Sandbox.Command.Result.t()

Run the suite against mutant mutant_id, wall-clock-timed, and return a typed Mutare.Sandbox.Command.Result.

test_args are extra mix test arguments ([] = whole suite, file-granular args otherwise); they are folded into the kill-detection argv by test_argv/1 (forcing --exit-status 101 and --max-failures 1). opts are the run options of Mutare.Sandbox.Command.Invocation.mix/4 — :cap bounds an overrun via the watcher, :max_heap_mb caps the heap, :partition carries the per-worker partition entry; [] sets none.