Mutation-test the current project.
Mutare builds a single metamutant, embedding every possible mutant behind a runtime switch, compiles it once, runs your test suite once as baseline with no mutants active, then for each mutant runs the part of the suite that potentially covers the mutant, reporting back with the mutants your tests failed to catch.
Getting started
Run it with no arguments to mutate everything under lib/. On a large project, start with a single module. It often is convenient to mutation test one module at a time, or otherwise slice-by-slice.
mix mutare # mutate everything under lib/
mix mutare --only lib/billing/invoice.ex # ...or just one file (a good first run)
mix mutare --max-mutants 50 # ...or just the first 50 mutants (to see mutare in action)The workflow is a loop: run it, look at each mutation your tests failed to catch—each survivor—then either add a test that would catch it or mark it with a # mutare:ignore comment if it is not worth testing, and run again.
Reading the results
Every mutant finishes in one of these states. The headline mutation score is the percentage of testable mutants your suite killed:
score = killed / (total − no_coverage − ignored − poisoned − harness_error)killed— a test failed on the mutant. Your suite caught the change.survived— every test still passed: no test tells the mutated code apart from the original. Survivors are the point of the tool, and print as a one-line diff so you can see exactly what slipped through.timeout— the mutant ran past the per-mutant time cap (e.g. it created an infinite loop). Counts as killed. (So does a mutant that exhausts the atom table and crashes the VM.)no_coverage— no test runs that line at all, so nothing could catch it. Excluded from the score; fix it by covering the line.ignored— suppressed by a# mutare:ignorecomment (below). Excluded.poisoned— the mutated code would not compile, so it was dropped. Excluded. (Rare — the built-in mutators are compile-safe.)harness_error— the mutant's test run never reached a pass/fail verdict (an infrastructure hiccup, not a real result). Excluded.
A run exits 0 even when mutants survive — survivors are findings to act on, not a build failure. See "Continuous integration" below to make a low score or a stale ignore exit non-zero.
Troubleshooting baseline-only failures
Mutare's baseline run executes the rewritten metamutant with no mutant active. For ordinary calls it should behave like your original code, but one implementation detail is observable: structural mutations that cannot be selected in-place — guards, head patterns, and clause-shape changes — are delivered by lifting the original function body into a generated dispatcher and leaving a wrapper at the source function name. If that code raises, exact exception metadata or stacktrace frames may name an internal function such as __mutare_name_arity_g1 instead of the original function.
A test that asserts FunctionClauseError.function, FunctionClauseError.arity, or exact stacktrace frame names can therefore pass under plain mix test and fail only inside Mutare's baseline. Prefer asserting the observable error and module/behavior, not the rewritten internal function identity.
If you need a compatibility escape hatch while changing those tests, configure skip_lifting: [{MyApp.Mod, :fun, arity}] or pass --skip-lifting MyApp.Mod.fun/arity. That keeps the matching function in-place, which also means Mutare will not generate guard, head-pattern, or clause-drop mutants for that function.
Suppressing a mutant
Some survivors are equivalent mutants — the mutation cannot change observable behaviour, so no test could ever kill it — or are simply not worth a test. Silence one with a # mutare:ignore comment at the end of the line (or on the line just above it):
def to_float(n), do: n * 1.0 # mutare:ignore multiplying by 1.0 is identityAfter the keyword come two optional, ordered parts — a [family] filter and a free-text reason:
# mutare:ignore suppress every mutant on the line
# mutare:ignore the reason text suppress all; record the reason
# mutare:ignore[arithmetic, relational] suppress only those families
# mutare:ignore[relational:>] suppress one kind: the `i > j` swap
# mutare:ignore[integer] off-by-one is ok a filter and a reason togetherThe names inside [...] are mutator families (see below). A family may be qualified with :label to suppress only one kind of its mutants — relational declares > >= < <= == != === !==, return_value declares empty/sentinel, integer declares zero/succ/pred, boolean declares negate. Run --list-mutators to see every built-in family's labels. Filtering fails safe: an unknown family, an empty [], or a malformed [… (no closing bracket) matches nothing, so the mutant runs rather than hides — but a qualified label a known built-in (or active custom) doesn't declare is a hard error (with a "did you mean"), so a typo can't silently fail to match. An ignore that suppresses no mutant (a typo'd family, a line that has no mutant) is reported as a warning — and with --strict-ignores, exits the run 1.
For a span that isn't worth annotating line by line — a literal lookup table, a generated module — two scoped verbs take the same filter and reason:
# mutare:ignore-start spot-checked; the round-trip test covers the table
def encode(?A), do: ?B
def encode(?B), do: ?C
# mutare:ignore-end
# mutare:ignore-file generated by `mix gen.tables` — do not hand-editA region suppresses everything from its -start through its -end (both delimiter lines inclusive); # mutare:ignore-file, anywhere in a file, suppresses the whole file. A broken pairing — an -end with no -start, nested -starts, a region never closed — is a hard error, and a scoped directive that suppresses nothing is warned exactly like a line one. Grammar details are in the Mutare.Ignore docs; audit what's suppressed with --list-ignores.
A directive is per line or per span. To leave a call alone everywhere it appears — an analytics emitter, a logger — route it instead: --skip-call Mixpanel.track/3, or a call_routes: entry in .mutare.exs (which can also leave single arguments as written, or keep a DSL macro's body out of the mutation set). See the configuration file section below and the README's "Routing calls".
Mutator families
All families run by default. Run mix mutare --list-mutators to print the catalog. Select a subset with --mutators a,b,c (or the :mutators key in .mutare.exs); list builtins to keep the whole default set and add to it — --mutators builtins,relational is every built-in, while --mutators relational is only the relational family. The family atoms, by kind:
- Operators —
arithmetic,operand_swap,bitwise,relational,strict_equality,logical,list,conditional - Literals —
integer,boolean,string,float,atom,convention,charlist,word_list,string_sigil,map,tuple,bitstring,bitstring_spec,regex,datetime,alias - Calls (rewrite or drop a stdlib/remote call) —
collection,collection_arity,string_call,string_byte,map_keyword,keyword_delete,map_set,period_boundary,call_removal,default_drop,mode_swap,numeric,math,integer_call - Structural —
return_value,if_condition,pattern_swap,pattern_wildcard,rescue_type,guard_drop - Behaviour-aware —
genserver(swaps an OTP callback's return tuple; fires only inside a@behaviour GenServermodule)
Each family's exact swap table lives in its own module's docs — print one with mix mutare --explain relational. You can also list your own module implementing Mutare.Mutator under :mutators to add a custom mutator.
Inspecting without running
These flags print information and exit, touching neither the sandbox nor the suite — for discovery, scripting, and debugging configuration:
mix mutare --version # the installed mutare version
mix mutare --list-mutators # the built-in mutator catalog (see above)
mix mutare --explain relational # one family's full documentation
mix mutare --list-macros # macros whose arguments are routed
# specially (built-ins + your config)
mix mutare --list-ignores # every `# mutare:ignore` in scope, each
# flagged active or ineffective
mix mutare --show-config # the effective options after merging
# .mutare.exs, CLI flags, and defaults
mix mutare --dry-run # list the mutants that *would* run, by
# file — no compile, no tests. Honours
# --only/--since/--mutators/--line/etc.
mix mutare --check # compile the metamutant (with poison
# recovery) but run no tests — a fast
# preflight for "will my DSLs build?".
# Prints a copy-pasteable :call_routes
# fix for any unknown macro it had to skipChoosing what to mutate
mix mutare path/to/project # target a different project directory
mix mutare --only lib/billing # scope to a directory
mix mutare --only lib/a --only lib/b # ...or several paths (repeatable)
mix mutare --exclude "lib/generated/**" --exclude lib/legacy
# skip files matching globs (repeatable)
mix mutare --since master # only lines changed vs a git ref
mix mutare --line lib/billing/invoice.ex:42
# only the mutants on that file:line — a
# narrow rerun, e.g. to recheck one
# survivor (repeatable; FILE:LINE is the
# exact prefix the report prints)
mix mutare --mutators relational,arithmetic # only some families (see above)
mix mutare --skip-lifting MyApp.Mod.fun/2
# keep one function in-place; no guard,
# head-pattern, or clause-drop mutants
mix mutare --skip-call Mixpanel.track/3
# skip every call to it — nothing inside
# the call is mutated (repeatable; also
# Mod.fun for any arity, Mod.* for a module)
mix mutare --no-expand-uses # don't expand `use` to discover the
# import/alias/@behaviour it injects
# (on by default; matters for Phoenix/Ecto)
mix mutare --no-seed-app-build # force a cold compile instead of reusing the
# app's built beams on a narrowed run
# (on by default)--only/--exclude/--line paths (and :paths in .mutare.exs) are resolved relative to the target project, not the directory mix was invoked from: targeting another checkout is mix mutare ./phoenix --only lib/phoenix/naming.ex — not --only phoenix/lib/.... A path that matches nothing aborts with no mutation sites found.
Continuous integration
By default a run exits 0 no matter how many mutants survive. CI gates add a non-zero exit when a run violates the policy you choose:
mix mutare --min-score 70 # exit 1 if the mutation score is below 70%
mix mutare --max-no-coverage 0 # exit 1 if any mutant has no covering test
mix mutare --fail-on-poisoned # exit 1 if any mutant had to be dropped
# because the mutated code would not compile
mix mutare --fail-on-harness-error # exit 1 if any mutant run reached no verdict
mix mutare --strict-ignores # exit 1 if any `# mutare:` comment matched
# no mutant (a typo'd verb/family or stale line)Combine --since with CI gates to gate only the code a pull request changed, and --quiet to drop the live progress animation (spinner, phases, per-survivor lines); the final report (and any machine reports) will still be printed.
mix mutare --since origin/main --min-score 80 --quietTuning the run
mix mutare --workers 8 # run 8 mutants concurrently (default: half
# your schedulers, capped at 4 — each worker
# is a full `mix test` BEAM that uses all
# of them)
mix mutare --full # run the whole suite for every mutant
# (default: only the test cases that cover it)
mix mutare --per-file # run whole covering test *files*, not just
# the individual covering tests — the opt-out
# for stateful async:false suites where
# per-test narrowing could hide a kill
mix mutare --no-full # force coverage-guided selection even if
# .mutare.exs set test_selection: :full
# (--no-per-file likewise restores :tests)
mix mutare --timeout 30000 # per-mutant wall-clock cap, in ms
# (default: derived from the baseline run)
mix mutare --timeout-multiplier 5 # ...or set the cap to baseline × this,
# scaled by half the concurrent workers
# (the baseline is timed uncontended)
# (default: 3.0; ignored if --timeout is set)
mix mutare --probe-timeout 600000 # wall-clock cap for the coverage probe run,
# in ms (default: 10× the per-mutant cap);
# an overrun degrades to run-all selection
mix mutare --max-heap-mb 4096 # cap each BEAM process's heap (in MB) in the
# baseline/probe/mutant runs — a mutation can
# make code allocate without bound (faster
# than the time cap can catch), and a capped
# runaway dies as an ordinary test failure
# instead of OOMing the machine. Size it well
# above the suite's biggest honest process;
# the baseline runs under the same cap, so a
# too-small value fails fast, up front
# (default: no cap)
mix mutare --baseline-runs 2 # run the green baseline 2× and abort if a
# test flakes (passes once, fails once) —
# a flaky test manufactures false kills
mix mutare --baseline-retries 3 # retry an all-red baseline up to 3× before
# aborting; mixed pass/fail baseline runs
# still abort as flaky
mix mutare --kill-runs 2 # require each killed mutant to kill twice;
# a passing rerun is reported survived
mix mutare --no-confirm-timeouts # record a timed-out run as :timeout right
# away. By default a timeout is confirmed
# with one sequential (uncontended) re-run
# first — the cap is derived from an
# uncontended baseline, so under parallel
# workers a slow-but-finite mutant could
# otherwise be falsely recorded as killed
mix mutare --harness-retries 4 # re-run a mutant up to 4× if its run fails
# at the infrastructure level (default 2)
mix mutare --max-harness-error-rate 0.3
# abort if more than 30% of the mutants
# that ran failed at the infrastructure
# level (1.0 = never abort on these)
mix mutare --max-survivors 5 # stop the run once 5 mutants have survived
# (in source order) — surface a few test
# gaps to fix without a full run. The result
# set is then partial, so CI gates are skipped
mix mutare --time-budget 10m # stop launching new mutants once 10 minutes of
# the per-mutant phase elapse (units h/m/s,
# e.g. 90s, 1h30m), draining the in-flight ones.
# "see what I can get in 10 minutes"; like
# --max-survivors the result set is partial, so
# CI gates are skipped
mix mutare --verbose # narrate what's happening at each step: a
# line per mutant (with its duration) plus
# per-phase detail — compile time, baseline
# timing, coverage breakdown, timeout cap,
# worker count. (--quiet wins over it)Flaky tests: expect mutation testing to find them
A mutation run re-runs your suite (or coverage-selected slices of it) far more times, in far more configurations, than normal CI does — so it is disproportionately good at surfacing pre-existing flaky tests. A flake at the baseline blocks the whole run (baseline suite is not green); one mid-run can manufacture a false kill. If your suite occasionally fails on its own, fix that first (or detect it explicitly with --baseline-runs 2) — it's a property of the target suite, not a Mutare failure to debug.
A related, narrower instability: a mutant on a timeout-shaped configuration literal (timeout: :infinity, a generous deadline) may be observable only under load — its verdict can honestly differ between runs because the mutated timeout only fires when something is slow. For codebases with such literals, --kill-runs 2 requires every kill to be reproduced, surfacing that load-dependence instead of recording whichever verdict the first run happened to produce.
Umbrella projects
mix mutare apps/billing # mutate one app (copies the whole umbrella)
mix mutare --app billing,web # specific apps (repeatable, and comma-separated)
mix mutare --app billing --app web # (equivalent to the line above)
mix mutare --workspace # mutate every app in the umbrellaDatabase isolation across workers
A suite with shared state (a database, say) can collide when several mutants run at once. --partition-db (or --partition-env <NAME> for a custom variable) gives each of the --workers concurrent runs a distinct partition id (1..workers) under an environment variable — MIX_TEST_PARTITION by default — so each worker can point at its own database:
mix mutare --workers 4 --partition-db # distinct MIX_TEST_PARTITION per worker
mix mutare --workers 4 --partition-env MY_SLOT # ...under a custom variable name
mix mutare --no-partition-db # disable a partition_env from .mutare.exsThis is the same convention as mix test --partitions, so a project already set up for that needs no code change:
# config/test.exs
config :my_app, MyApp.Repo,
database: "my_app_test#{System.get_env("MIX_TEST_PARTITION")}"You must pre-create and migrate the --workers partitioned databases (just as is required by mix test --partitions). The pool recycles ids across the run, so --workers 4 needs four databases, not one per mutant; the baseline and coverage probe use partition 1.
Sandbox and build cache
mix mutare --sandbox /tmp/mut # put the sandbox at a path of your choosing (inspect it, or cache it on CI)
mix mutare --no-keep-sandbox # throwaway sandbox: rebuild cold, remove it afterwardsBy default Mutare keeps the sandbox between runs (--keep-sandbox, on): it lives at a stable per-project temp dir and is re-materialised incrementally — only changed files are rewritten, so mix's compiler reuses the cached _build and a re-run after adding a test recompiles nothing. That is the compile-once bet applied across runs, not just within one; the cost is one project copy plus its build sitting in the temp dir per project. --sandbox <path> chooses the location — handy for inspecting the generated metamutant, and on CI point it at a cached directory (cache <path>/_build and <path>/deps, keyed on mix.lock; tar-based caches preserve the mtimes the incremental compile relies on). --no-keep-sandbox opts out: a fresh copy, a cold compile, and the sandbox removed when the run finishes. Reach for it when a kept sandbox has gone bad — a build artifact corrupted by a crashed run, say, shows up as every mutant reporting a harness error — since the wipe is the reset.
Output formats
mix mutare --report json:mutare.json
# write a machine report to a file; the
# human report still prints to the console
mix mutare --report sarif # emit SARIF to stdout (this suppresses the
# human report, so the two don't collide)
mix mutare --report json:mutare.json --report sarif:mutare.sarif
# `--report` is repeatable; omit :PATH to
# write that report to stdout--report takes FORMAT[:PATH], where FORMAT is one of human (the default console report), json (the mutation-testing-elements / Stryker report schema), html (that JSON embedded in the interactive report viewer), or sarif (survivors as findings for GitHub code scanning).
Each destination takes one report: at most one may omit :PATH (two documents on stdout would be valid in neither format), and no two may name the same path (only the last written would survive). Either collision is a startup error naming the clashing formats.
Configuration file (.mutare.exs)
Configuration may also live in .mutare.exs (a keyword list); a CLI flag overrides the matching key. Every option is optional — the block below lists all the file-settable keys with their defaults:
# .mutare.exs
[
# --- what to mutate ---
paths: ["lib"],
exclude: ["lib/generated/**"],
# built-in family atoms and/or your own Mutare.Mutator modules. The
# `:builtins` token means "all built-ins", so `[:builtins, MyMutator]`
# extends the defaults and `[MyMutator]` replaces them;
# `{:builtins, except: [:arithmetic]}` drops a family. Omit the key for
# the full default set.
mutators: [:builtins],
# skip a call outright (`:skip` — an analytics emitter, a logger), or leave a
# macro's arguments as written (`:raw` — a DSL body, a pattern) so they
# aren't mutated; a list treats each position (`:expression`, `:raw`,
# `:interior`, a keyed `[timeout: :raw]` refinement of a keyword argument);
# `:*` wildcards a slot: {M, :*, :skip} = whole module, {:*, name, :raw}
# = that name in any module (a more specific line overrides)
call_routes: [{Mixpanel, :track, 3, :skip}, {Ecto.Query, :from, :raw}],
# extend the built-in timeout table (and any label a companion package
# documents) to your own functions: {Module, :fun, arity, positions, label}
argument_marks: [{MyApp.Http, :get, 2, [{:keyword, :recv_timeout}], :timeout}],
# keep specific functions in-place when lifted function names are observable;
# entries are {Module, function_name_atom_or_string, arity}
skip_lifting: [],
# non-mutating source-understanding modules implementing
# Mutare.CallRouting, Mutare.UseExpansion, or both
extensions: [],
# expand `use` to surface the import/alias it injects (--no-expand-uses)
expand_uses: true,
# --- how the suite runs ---
# :tests runs only the individual test cases covering each mutant; :coverage
# runs whole covering files (opt-out for stateful async:false suites); :full
# runs the whole suite for every mutant
test_selection: :tests,
# concurrent mutant runs; default: half the schedulers, capped at 4 (each
# worker is a full `mix test` BEAM that itself uses every scheduler)
workers: 4,
# give each concurrent worker a distinct partition id under this env var
# (1..workers), for per-worker DB isolation — read it in config/test.exs
# like `mix test --partitions`; nil (default) is off. Needs `workers` DBs.
partition_env: nil,
# per-mutant wall-clock cap = baseline run × multiplier × half the
# concurrent workers (the baseline is timed uncontended), unless an
# absolute `timeout:` in ms is given instead (then the multiplier is moot)
timeout_multiplier: 3.0,
timeout: nil,
# cap each BEAM process's heap (MB) in the baseline/probe/mutant runs, so a
# mutation that allocates without bound dies as an ordinary test failure
# instead of OOMing the machine; nil (default) is no cap
max_heap_mb: nil,
# run the baseline N×, aborting if a test flakes (passes one run, fails another)
baseline_runs: 1,
# retry a consistently-red baseline attempt before aborting (useful for
# target suites with occasional startup/load flakes)
baseline_retries: 0,
# require a killed mutant to kill N times before recording the kill; if any
# rerun passes, record it as survived (unanimous-kill, default unchanged)
kill_runs: 1,
# retry a mutant whose run fails at the harness (infra) level before recording it
# (a boot-time node crash, a known-transient contention signature, is retried
# harder still from its own dedicated budget — see `mix help mutare`)
harness_retries: 2,
# abort if more than this fraction of the mutants that ran erred at the
# harness level (1.0 = never abort on harness errors)
max_harness_error_rate: 0.5,
# test at most the first N mutants in source order (a quick smoke run)
max_mutants: nil,
# stop the run once the first N surviving mutants are found (an
# iterate-and-fix workflow); the partial result set skips CI gates
max_survivors: nil,
# stop launching new mutants once this much wall-clock time in the
# per-mutant phase elapses — a duration string like "10m"/"90s"/"1h30m"
# (nil = no budget); in-flight mutants drain, and the partial result set
# skips CI gates, exactly like max_survivors
time_budget: nil,
# --- sandbox reuse / build cache (see "Sandbox and build cache" above) ---
sandbox: nil,
# keep the sandbox (and its compiled _build) between runs, re-materialising
# it incrementally; false = throwaway copy, cold compile, removed afterwards
keep_sandbox: true,
# on a narrowed run (--only/--line/--since), reuse the app's already-built
# beams so the one compile rebuilds just the mutated file(s) — not the whole
# app. On by default; --no-seed-app-build forces a cold compile
seed_app_build: true,
# --- output & CI gates ---
# exit the run with code 1 if the mutation score drops below this percentage; by default there is no minimum score
# min_score: 70,
# exit 1 if the run records more than this many uncovered mutants; nil disables the gate
max_no_coverage: nil,
# exit 1 if any mutant had to be dropped because the mutated code would not compile
fail_on_poisoned: false,
# exit 1 if any mutant's test run reached no pass/fail/timeout verdict
fail_on_harness_error: false,
# exit 1 if any `# mutare:ignore` suppresses no mutant (a typo or stale
# line), or any `# mutare:` comment names no recognized directive
strict_ignores: false,
# suppress the live stderr progress (for CI / piped use)
quiet: false,
# narrate each step in detail: a line per mutant + per-phase numbers
# (compile/baseline timing, coverage breakdown, cap, workers). `quiet` wins
verbose: false,
# emit several reports at once (default is [:human]; the file path is optional and if omitted the report is printed to stdout.)
# reporters: [:human, {:json, "mutare.json"}, {:sarif, "mutare.sarif"}]
]