An ExUnit formatter that writes a machine-readable per-test report.
ExUnit has no such output, and mix verify.change's quarantine ceiling needs
one: the count of tests excluded by the :quarantine tag is the number the
ceiling bounds, and stdout scraping cannot distinguish that exclusion from
the several others this suite already applies (:playwright,
:integration_caddy, :integration_cli).
Why it lives in ex_sandbox, and why in lib/
A consumer with several applications needs one formatter every one of their suites can load, and the only module that qualifies is one they all already depend on. This library is that module for its own consumers, which is the whole of the reason — nothing here is about sandboxing, and nothing here names any particular application.
⚠️ lib/, and it was test/support/ until 1.0.1. Inside an umbrella that
worked, because test/support is on elixirc_paths under MIX_ENV=test and
the compiled module landed in _build/test/lib/ex_sandbox/ebin where every
sibling app's mix test could load it. A published package has no such
arrangement: test/ is not in package/0's files:, so 1.0.0 listed this
module as Public in priv/boundary.md and shipped a release that does not
contain it. The consumer's own --formatter ExSandbox.LoopFormatter then
fails to load a module its dependency promised.
Why TSV rather than JSON
ex_sandbox depends on :telemetry and nothing else -- no JSON encoder is
available here, and adding one to the dependency root of the umbrella to
format a test report is not a trade worth making. The consumer is a bash 3.2
runner with awk, which reads TSV natively.
The last line is the point
The report ends with a summary row whose final field is COMPLETE. A run
that crashes, is killed, or fills the disk leaves a report without it.
This matters more than it looks. A gate that reads a truncated report sees
fewer failures and fewer quarantined tests than actually exist, and passes --
reporting green because it was handed less evidence, which is the exact
failure docker/census-gate.sh was written to close and the one this
repository has been bitten by three times. mix verify.change therefore
treats a report without a trailing COMPLETE as a gate failure, never as an
empty result.
Enabling
Set LOOP_REPORT to the destination path. Unset, the formatter records
nothing and costs nothing, so it is safe to leave configured.
LOOP_REPORT=.loop/runs/report.tsv mix test --formatter ExSandbox.LoopFormatterRows are appended, because an umbrella mix test runs each app in its own
ExUnit process and this formatter is initialised once per app. The consumer
sums across every summary row and requires the file's last row to be one.
Summary
Functions
Returns a specification to start this module under a supervisor.
Functions
Returns a specification to start this module under a supervisor.
See Supervisor.