Mutation testing: change the production code, and see whether the suite notices.
A passing suite says the tests ran. It does not say they guard anything.
The only cheap way to tell the difference is to break the code on purpose and
check that something goes red — and every place that has been done by hand in
this project has found real gaps: a record(...) that could be deleted from a
frame tracker with 1500 tests still green, an -export line whose removal
crashes every app at boot and which no assertion noticed, a requested:
option whose deletion silently restored the bug it was added to fix.
Mob's native assertions make this sharper than usual. They match source text
with =~, which is unusually easy to write vacuously — an assertion satisfied
by the comment describing the code, or one pinning a rename rather than a
behaviour. Mutation is what separates those from the real ones.
Known limits
Deleting a line is blunt on Elixir: removing a def head or a middle segment
of a pipeline is a syntax error, not a test failure. Those are reported as
did not build and counted apart from real kills, because a mutant that
cannot compile says nothing about the tests. Expect roughly a third of the
mutants on idiomatic Elixir to land there.
Mutating a module that itself manipulates source text — this one — produces self-referential noise, since its operator table contains the very tokens it searches for. That is a curiosity here rather than a problem in general.
Nothing understands strings: a " == " inside a literal is mutated like any
other. Skipping heredocs and type attributes removes the bulk of that, and
the rest is cheap enough to read past.
Why this is a task and not a shell loop
It has been a shell loop, six or seven times, and one of those runs compared the mutant's output against a hardcoded expected pass-count that was off by one. Every surviving mutation in that run was reported as caught, including a real gap. Capturing the baseline by running it is the whole difference, and it is not something to re-derive under time pressure at the end of a task.
Summary
Types
One change to try.
Functions
Line numbers diff adds or changes, per file.
Classify one mutant's outcome by comparing against the captured baseline.
Whether a line is worth mutating.
The mutations worth trying for source, restricted to lines when given.
Line numbers inside a """ block, which are prose or data rather than code.
The Result: line ExUnit prints, or nil when it printed none.
Types
Functions
@spec changed_lines(String.t()) :: %{required(String.t()) => MapSet.t(pos_integer())}
Line numbers diff adds or changes, per file.
Reads a unified diff and follows the @@ hunk headers, counting only the
lines present in the new file — a deleted line has no line to mutate.
@spec classify(String.t(), String.t(), String.t() | nil) :: :survived | :killed | :build_error | :unmeasured
Classify one mutant's outcome by comparing against the captured baseline.
baseline must be the output of actually running the suite on unmutated
code. Comparing against a remembered or hardcoded value is the mistake this
module exists to stop making: a baseline that is wrong by one test reports
every survivor as killed.
:survived— identical result, so nothing noticed the change.:killed— the suite went red, which is what should happen.:build_error— it did not compile. The mutant died, but that says nothing about test quality, so it is reported separately rather than counted as a win.:unmeasured— the run printed no result and named no build error we recognise. Counting that as a kill is the same score inflation in a quieter form: the safe answer to "I could not measure this" is to say so.build_error?/1matches four banners, and Mix has more than four ways to fail.
Whether a line is worth mutating.
Blank lines, comments and bare closing delimiters are skipped. Deleting an
end or a } is a guaranteed syntax error, which the runner would report as
killed — technically true and completely uninformative, and enough of them
drown the survivors that matter.
@spec mutations(String.t(), String.t(), MapSet.t(pos_integer()) | :all) :: [ mutation() ]
The mutations worth trying for source, restricted to lines when given.
lines is the set of 1-indexed line numbers to consider — normally the lines
a branch changed, so a run reports on the work in hand rather than the whole
repository.
@spec prose_lines([String.t()]) :: MapSet.t(pos_integer())
Line numbers inside a """ block, which are prose or data rather than code.
Found by running this tool on itself: without it, every line of a @moduledoc
is offered as a deletion, every one survives — nothing asserts on prose — and
a report of ten survivors contains ten pieces of noise and no signal. The
delimiter lines go too, since deleting one unbalances the block and only ever
produces a syntax error.
A heredoc is the common case in Elixir; Swift's multi-line literals use the same delimiter, so the same scan covers both.
The Result: line ExUnit prints, or nil when it printed none.
nil means the run did not get as far as reporting, which the caller must
treat as a failure to measure rather than as a passing baseline.