CONFORMANCE — the harness and the fuse (v1, frozen)

Copy Markdown View Source

The conformance harness is a first-class artifact, not a test appendix. It is the mechanism that makes "LiveView node continuity is observed, not guaranteed" (SPEC.md §5) an actively-monitored contract instead of a footnote.

What it is

Playwright drives real Chromium/Firefox/WebKit against real Phoenix LiveView apps over a real LiveSocket/WebSocket. It observes DOM node-object identity (JS expando, not HTML string equality) and each machine's state at its native source (:popover-open, dialog.open/open attribute, document.activeElement, scrollTop, aria-activedescendant). Every case runs twice per engine and the runs must be byte-identical.

Harnesses:

  • harness/ — delegation ledger (popover, dialog, details, select, focus, scroll).
  • cursor-spike/ — client-owned tier (replace-with-same-id).
  • combobox-spike/ — composition (Delegation × Cursor × Channel).

The focused Aha trial

The main harness also compresses the ownership result into one causal comparison:

  1. open a manual popover and apply an acknowledged aha_rev patch;
  2. on a fresh page, open a raw modal dialog and an otherwise-identical dialog whose open attribute alone is protected by JS.ignore_attributes(["open"]);
  3. apply the same acknowledged patch shape and read the complete native state vectors.

The raw and protected dialogs are the controlled pair; popover is the native-safe contrast. The assertion requires all three DOM node objects and server-rendered child updates to survive. It then distinguishes a coherent machine from the observed raw dialog tear (open=false, open attribute absent, :modal=true, no lifecycle event). This is deliberately stronger than an open boolean and does not confuse a stable DOM ID with object identity.

Multi-version

The target app selects the LiveView version via LV_SPEC (hex version, github:owner/repo#ref, or path:). The harness records Phoenix / LiveView / Elixir / OTP / browser versions with every result set.

The fuse — two-tier gate semantics (frozen)

Tier 1 — stable LiveView: blocking gate.

  • green-machine regression → fail (blocks PR / release)
  • red fixtures → expected failure; a red turning green also fails (records a behavior change to reconcile)
  • unknown / amber → do not block, but MUST appear in the regenerated ledger

Tier 2 — LiveView main/edge: early-warning fuse.

  • runs at least on a schedule, and on manual dispatch
  • failure makes the run fail and uploads the result artifact, but is non-blocking for ordinary PRs initially
  • purpose is not to promise support for unreleased versions — it is to catch a change in observed behavior before it ships
  • a Tier-2 failure surfaces in the ledger as future risk, never silently skipped

Matrix sizing.

  • PR workflow MAY run a reduced browser matrix for speed.
  • The release workflow MUST run the full Chromium/Firefox/WebKit matrix.
  • The scheduled/full workflow runs the full matrix against both stable and main.
  • The cursor and combobox harnesses run as their own jobs in the same workflow.

The assertion

harness/ci-assert.mjs reads harness/results.json and enforces the frozen classification: green machines must be green on all engines; red fixtures must be red. It also gates the focused Aha state vectors on both runs of every selected engine. It exits non-zero on any violation. harness/ledger.mjs regenerates delegation-ledger.{md,json} from the same results so the ledger is CI-regenerated, never hand-maintained.

Consumer runner release gate

The external-app runner has a separate gate because calibration evidence cannot prove that a published consumer package is complete or that its four verdicts are classified correctly:

  • npm run test:contract-runner drives a local fixture through pass, fail, inconclusive, and error paths, including single-contract compatibility, suite preflight/reduction/run-all behavior, navigation, and evidence-file safety.
  • test/package-smoke.sh unpacks the actual Hex artifact, compiles it as an isolated dependency, launches its bundled runner against a real browser, and checks the public Mix argument boundary.
  • the CI consumer job runs the public Mix suite mode against the real LiveView harness and requires coherent popover/protected-dialog passes plus the raw-dialog red control, each retained as an independent schema-v1 result.

All three checks are blocking. A repository-only pass is not sufficient for release.

Reproduction

See REPRODUCE.md.