Separates the suite's third outcome from failure in the exit status
(012-FR-016a).
The gap this closes
FR-016 is about reporting, and the suite already reports well: an
undemonstrable check prints NOT DEMONSTRATED (host capability unavailable)
and says what would turn it into a pass. What it could not do is make that
distinction reach a machine.
ExUnit has two outcomes. ExSandbox.Conformance.Group turns the third into a
failure — deliberately, because the alternative is reporting an
undemonstrated guarantee as green (FR-012b), and because ExUnit's skip tag
resolves before the test runs while ExSandbox.Capability answers only at
runtime. That decision is correct and this module does not revisit it.
Its consequence is what needed fixing. mix test exits non-zero for a
mechanism defect and non-zero for a host that cannot demonstrate six
credentials checks, and nothing downstream can tell the two apart. The
isolation container therefore exited 2 on every green run, so
--exit-code-from isolation carried no information: a genuine containment
regression would have looked exactly like the six third outcomes that are
supposed to be there.
What it does, and what it deliberately does not
It is an ExUnit formatter, which means it observes. It receives each
finished test, classifies the failures the suite already decided on, and
writes a census to EX_SANDBOX_CENSUS_PATH. It sets no verdict, rescues
nothing, and changes no test's outcome — mix test still exits non-zero, and
every third outcome is still printed in full by the default formatter running
alongside it.
The caller decides what to do with the census. docker/run-isolation-tests.sh
treats "failures, all of them third outcomes" as a pass and anything else as a
failure; that policy lives in the caller because it is a caller's judgement,
not a property of the contract.
Why matching on the message is sound here
Classification keys on the marker string in
ExSandbox.Conformance.Group.not_demonstrated/1, which is the single place
the framing is produced — check/2 and guarded_setup/1 both route through
it precisely so they cannot drift. ExSandbox.ConformanceExclusionsTest
asserts the marker still matches, so a reworded message fails a test rather
than silently reclassifying every third outcome as a defect.
⚠️ The failure direction matters and is chosen: an unrecognised message counts as a defect, never as a third outcome. If this module's matching ever breaks, the suite over-reports failures and someone investigates. The opposite default would quietly convert real violations into "unavailable" and hand back a green exit code, which is the precise artefact this suite exists to prevent.
Summary
Functions
Returns a specification to start this module under a supervisor.
Functions
Returns a specification to start this module under a supervisor.
See Supervisor.