Run the test suite until it fails, or a set number of times, and report which tests were not deterministic.
A flake does not announce itself. It fails once, on someone else's branch, and the natural response is to re-run and move on — which is how a 1-in-17 failure survived long enough to corrupt a mutation-testing verdict and send a bisect down the wrong path. This makes looking cheap and deliberate.
mix mob.flake # 20 runs, stop at the first failure
mix mob.flake --runs 50 # more attempts
mix mob.flake --until-failure # keep going until one fails
mix mob.flake --keep-going # run them all, report every failure
mix mob.flake test/mob/nav # narrow the target
mix mob.flake --seed 0 # fix the seed, so ordering is constantReading the result
A test that fails under --seed 0 on every run is not flaky, it is broken.
This looks for the other kind: tests that pass and fail with everything else
held still. Those are almost always a race — a Process.sleep standing in
for synchronisation, a check-then-act across a process boundary, or shared
global state (a named process, an ETS table) whose lifetime is tied to
whichever test happened to start it.
Narrowing helps more than volume. If a failure names one module, run that module 200 times rather than the suite 20 more times.