ExSandbox.Conformance.Census (ExSandbox v1.0.1)

Copy Markdown View Source

Separates the suite's third outcome from failure in the exit status (012-FR-016a).

The gap this closes

FR-016 is about reporting, and the suite already reports well: an undemonstrable check prints NOT DEMONSTRATED (host capability unavailable) and says what would turn it into a pass. What it could not do is make that distinction reach a machine.

ExUnit has two outcomes. ExSandbox.Conformance.Group turns the third into a failure — deliberately, because the alternative is reporting an undemonstrated guarantee as green (FR-012b), and because ExUnit's skip tag resolves before the test runs while ExSandbox.Capability answers only at runtime. That decision is correct and this module does not revisit it.

Its consequence is what needed fixing. mix test exits non-zero for a mechanism defect and non-zero for a host that cannot demonstrate six credentials checks, and nothing downstream can tell the two apart. The isolation container therefore exited 2 on every green run, so --exit-code-from isolation carried no information: a genuine containment regression would have looked exactly like the six third outcomes that are supposed to be there.

What it does, and what it deliberately does not

It is an ExUnit formatter, which means it observes. It receives each finished test, classifies the failures the suite already decided on, and writes a census to EX_SANDBOX_CENSUS_PATH. It sets no verdict, rescues nothing, and changes no test's outcome — mix test still exits non-zero, and every third outcome is still printed in full by the default formatter running alongside it.

The caller decides what to do with the census. docker/run-isolation-tests.sh treats "failures, all of them third outcomes" as a pass and anything else as a failure; that policy lives in the caller because it is a caller's judgement, not a property of the contract.

Why matching on the message is sound here

Classification keys on the marker string in ExSandbox.Conformance.Group.not_demonstrated/1, which is the single place the framing is produced — check/2 and guarded_setup/1 both route through it precisely so they cannot drift. ExSandbox.ConformanceExclusionsTest asserts the marker still matches, so a reworded message fails a test rather than silently reclassifying every third outcome as a defect.

⚠️ The failure direction matters and is chosen: an unrecognised message counts as a defect, never as a third outcome. If this module's matching ever breaks, the suite over-reports failures and someone investigates. The opposite default would quietly convert real violations into "unavailable" and hand back a green exit code, which is the precise artefact this suite exists to prevent.

Summary

Functions

Returns a specification to start this module under a supervisor.

Functions

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

See Supervisor.