CONFORMANCE — the harness and the fuse (v1, frozen)

Copy Markdown View Source

The conformance harness is a first-class artifact, not a test appendix. It is the mechanism that makes "LiveView node continuity is observed, not guaranteed" (SPEC.md §5) an actively-monitored contract instead of a footnote.

What it is

Playwright drives real Chromium/Firefox/WebKit against real Phoenix LiveView apps over a real LiveSocket/WebSocket. It observes DOM node-object identity (JS expando, not HTML string equality) and each machine's state at its native source (:popover-open, dialog.open/open attribute, document.activeElement, scrollTop, aria-activedescendant). Every case runs twice per engine and the runs must be byte-identical.

Harnesses:

  • harness/ — delegation ledger (popover, dialog, details, select, focus, scroll).
  • cursor-spike/ — client-owned tier (replace-with-same-id).
  • combobox-spike/ — composition (Delegation × Cursor × Channel).

Multi-version

The target app selects the LiveView version via LV_SPEC (hex version, github:owner/repo#ref, or path:). The harness records Phoenix / LiveView / Elixir / OTP / browser versions with every result set.

The fuse — two-tier gate semantics (frozen)

Tier 1 — stable LiveView: blocking gate.

  • green-machine regression → fail (blocks PR / release)
  • red fixtures → expected failure; a red turning green also fails (records a behavior change to reconcile)
  • unknown / amber → do not block, but MUST appear in the regenerated ledger

Tier 2 — LiveView main/edge: early-warning fuse.

  • runs at least on a schedule, and on manual dispatch
  • failure makes the run fail and uploads the result artifact, but is non-blocking for ordinary PRs initially
  • purpose is not to promise support for unreleased versions — it is to catch a change in observed behavior before it ships
  • a Tier-2 failure surfaces in the ledger as future risk, never silently skipped

Matrix sizing.

  • PR workflow MAY run a reduced browser matrix for speed.
  • The release workflow MUST run the full Chromium/Firefox/WebKit matrix.
  • The scheduled/full workflow runs the full matrix against both stable and main.
  • The cursor and combobox harnesses run as their own jobs in the same workflow.

The assertion

harness/ci-assert.mjs reads harness/results.json and enforces the frozen classification: green machines must be green on all engines; red fixtures must be red. It exits non-zero on any violation. harness/ledger.mjs regenerates delegation-ledger.{md,json} from the same results so the ledger is CI-regenerated, never hand-maintained.

Reproduction

See REPRODUCE.md.