Testing and Conformance

Status: Agreed

This document settles how conformance to the specification is demonstrated: the projection golden that already exists, and the surface gate, heading check, CI pipeline and formal model that are agreed but not yet built. Each section states whether it is implemented today.

1. The projection golden

Implemented. test/support/projection_golden.txt records Visualize.Geo.Projection.project/3 and invert/3 for all 38 projection types over a fixed 9 × 9 grid of longitudes [-170, -120, -60, -15, 0, 15, 60, 120, 170] and latitudes [-80, -60, -30, -5, 0, 5, 30, 60, 80], with scale(100) and translate(0, 0). Each line is type lon lat -> x y | ilon ilat, or -> nil when the forward projection clips, or | nil when the inverse does; the test renders the same text and asserts byte equality.

• Numbers are printed to nine decimals, far below the projections' own iteration tolerances, so a real change in the maths shows and float noise from reordering an expression does not.
• Negative zero is folded to positive zero before printing (D-6). A residual such as -4e-10 still prints as -0.000000000; that is a property of the maths, not of the sign of zero, and is stable.
• The inverse of :transverse_mercator is recorded as excluded pending issue #9 (D-5).
• A second test asserts that the golden's set of type names equals the module's list, so adding a projection without regenerating fails.

The golden MUST be regenerated (mix run -e 'Visualize.Geo.ProjectionGoldenTest.write!()') only when projection maths changes on purpose, and the regeneration MUST be committed with the change that caused it.

The clip on the sphere is held to d3-geo (#509, #510). test/support/geo/clip_golden.jsonl is d3-geo's own output — one JSON case per line, the projection in the library's terms (with its clip_angle where the case sets one; without one the type's default, so the defaults are held to d3's too), the GeoJSON input and d3's stream of subpaths, a ring inside a polygon and a line otherwise — for the cases of 07-geo §2.2.1: lines across the seam and over a pole, polygons across it, with a hole, around a pole and around all but one; lines and polygons across small circles of 60°, 120° and the orthographic's 90 + 10⁻⁶, through one from outside, around its centre and around the antipode, wound the other way and with a hole; and Natural Earth 1:110m land under rotated equirectangular and Natural Earth projections and under rotated, tilted and rolled orthographic, stereographic and azimuthal-equidistant ones, every projection at precision(0); and the rotation itself (#511): points projected under several [lambda, phi, gamma] through the equirectangular, Mercator, Natural Earth, Equal Earth and the five azimuthal types, with d3's projected point — null where d3's stream drops it — and d3's inverse of it. The projection is passed to d3 as [-lambda, phi, gamma]: phi and gamma are d3's own and only lambda keeps the library's opposite sign (07 §1.4, D-44, D-129). Its header names the d3-geo version and the command. It is written by test/support/geo/clip_golden.mjs, a one-off node script and, with the circle's oracle below, the suite's only JavaScript, kept because d3-geo itself is the reference: it runs by hand, against a d3-geo installed outside the repository, and the test never runs node. test/visualize/geo/clip_test.exs projects each case at precision 0 and compares every subpath with d3's point by point within 1e-6, and each rotation case's points — Visualize.Geo.Projection.project/3, nil where d3 drops the point, and, for every type but the Natural Earth and Equal Earth, whose inverses iterate to tolerances of their own, invert/3 of d3's projected point — within 1e-9; a ring without a closing vertex that repeats its first (d3 drops it where the clip leaves the ring whole, the library keeps it). It is regenerated only when the clip changes on purpose or d3-geo is moved to a later release.

The geo circle is held to d3-geo (#525). test/support/geo/circle_golden.jsonl is d3-geo's geoCircle itself — one JSON case per line, the circle's center, radius and precision and d3's ring, every number as JSON.stringify writes it, the shortest that reads back as the same double — for the centres, radii and precisions of 07-geo §2.5, and its header names the d3-geo version and the command. It is written by test/support/geo/circle_golden.mjs, the clip oracle's one-off companion, run by hand against the same d3-geo installed outside the repository; the test never runs node. test/visualize/geo/circle_test.exs builds each case with Visualize.Geo.Circle.polygon/1 and compares the ring with d3's position by position within 1e-9 degrees, the lengths equal. It is regenerated only when the port changes on purpose or d3-geo is moved to a later release.

2. The API surface golden

Implemented. API_SURFACE.md at the project root is generated by mix vis.surface --write and verified by mix vis.surface --check. It joins two sets: every `Module.function/arity` found in the first cell of a table row under spec/, and every public (non-@doc false) function defined in lib/, excluding Visualize.MixProject. A function a module injects through use is not in the source the scanner reads, so a module that injects functions declares what it injects through a zero-arity, undocumented export and the scanner asks it: the builder's schema-generated functions (14-declarative-chart §16.6) are rows of the golden like any other.

Each row is {Function, Section, Status, Version, Locus}:

• Function — the fully qualified Module.function/arity.
• Section — the NN-slug §n.m heading that declares it, or empty.
• Status — specified (declared and defined), unspecified (defined in lib/, no row in spec/), unimplemented (row in spec/, no definition in lib/).
• Version — a content hash of the function's definition, all clauses together, so a row changes only when the definition changes and unrelated rows never conflict on merge. It is empty for an unimplemented row, which has no definition to hash.
• Locus — the source file that defines it, relative to the project root, and empty for an unimplemented row.

--check MUST fail when the regenerated file differs from the committed one or when any row is not specified, and MUST name the rows that are not, with the remedy for each: write the row for an unspecified function, build it or delete the row for an unimplemented one. It is the HARD gate api-surface-drift: it cannot be waived per-merge, only changed by editing the specification. mix vis.surface with no flag reports the same two backlogs without failing, which is what makes it the form to run while the work is in hand.

3. Heading numbering

Implemented. mix vis.headings reads every spec/*.md and fails when: a file has more than one # title; line 3 is not a **Status:** line with one of the three vocabulary words; any ##, ### or #### heading lacks a decimal number; numbers at any level are not gap-free from 1; or a number is duplicated. It is what makes §n.m references in API_SURFACE.md stable.

4. The CI pipeline

Implemented (.gitlab-ci.yml). Three stages, each blocking the next:

The two warning gates cover disjoint code and both are required (D-91). mix compile --warnings-as-errors compiles what elixirc_paths/1 lists — lib, audit, specs and test/support — and never reads a *_test.exs file, so a warning in a test is invisible to it; mix test --warnings-as-errors is the one that fails on a warning raised while compiling the suite itself, and it exits non-zero after the tests have run rather than before them, so a warning is reported beside the results and not in place of them.

An optional dependency runs both ways. The check stage has every optional dependency, so the suite runs each feature with its package; the consumer-compile job builds a consumer that lists none of them, so it runs each feature's absent branch.

The resvg binary runs both ways too (D-121). PNG output needs a binary on the host, not a package (09-rendering-backends §9.4):

• The test job downloads a pinned release. Its script installs curl, downloads https://github.com/linebender/resvg/releases/download/v0.48.1/resvg-linux-x86_64.tar.gz, checks it against the SHA-256 written in .gitlab-ci.yml (fa8c26495a187e592c501db15bf9e8a9fdc051d4b2b336b39703d5b59f912b9d) with sha256sum -c, extracts the one file resvg into .resvg/, and runs the suite with VISUALIZE_RESVG set to it. A tarball that does not match fails the job before any test runs. Moving to another resvg release is a change to the URL and the checksum together, and the raster goldens are regenerated with it when its pixels move beyond the tolerance (§6).
• test/test_helper.exs reads VISUALIZE_RESVG. When it is set, the helper configures config :visualize, :resvg with it, and a value that names no executable file stops the suite with an error rather than skipping: a pipeline that meant to rasterise never silently does not. When it is unset, the library's own lookup applies, resvg on the PATH. When no binary is found either way, the tests tagged :resvg (every test that runs the binary) are excluded, and the helper says so in one line, as it does for :js without node.
• The absent case runs everywhere. The {:error, :no_rasterizer} branch of both to_png/3 and to_png!/3 is reached through the availability argument, untagged, so it runs with or without a binary; and the consumer check runs to_png/2 on a host with none.
• The CI image has no system fonts, so every raster test that draws text renders with the bundled test font and system_fonts: false: the raster goldens of §6 are the same on every host.

mix vis.check (scripts/check.exs) runs the same jobs locally in the same order and MUST be the last thing run before submitting a change for review. Its verify:hooks runs mix test --only js when node is on the path and otherwise prints verify:hooks: skipped: no node and goes on, so a workstation without node stays green and the pipeline is where the hooks are always executed. The raster tests follow the same rule through the :resvg tag: mix vis.check uses the binary VISUALIZE_RESVG names or the PATH holds, and without one the test job prints the helper's line and stays green.

Compiled dependencies come from the depdep store (scripts/depdep.exs, pinned to a depdep tag; #342, #343). Every job's before_script pulls the root project's dependencies in one line — --pull --mix-get --compile-deps --exclude examples: the pull, mix deps.get inside it, a second decision once the source is on disk (so a git dependency and its cone are keyed in one invocation), and a compile of exactly what the store lacked — falling back to a bare mix deps.get when the store cannot be reached; the compile job pushes what was missing after a green build, the examples member excluded since those jobs never build it. The examples:test job is the one that builds examples/, so it pulls and pushes that member itself, from its own directory, the same way. Without the DEPDEP_* variables depdep is a no-op that exits 0, so a pipeline never depends on the store.

5. The ExTLA model of the force simulation

Implemented. Visualize.Specs.ForceSimulation (specs/force_simulation_spec.ex, compiled in dev and test only) models the timer discipline of Visualize.Layout.Force.Simulation (06-layouts §8.5). Nodes, links and forces are abstracted away; what is kept exactly is every message the server handles, the deferred :start_simulation that init/1 enqueues, and the two kinds of tick message: a live tick from the timer the server is waiting on, and a stale tick from a timer cancelled after its message had already reached the mailbox.

The checker is extla, a dev and test dependency pinned to a release tag with its mix.lock entry committed (#441), so an extla change reaches this project only by a deliberate bump. CI and every workstation build the same extla. Building against a local checkout is an explicit opt-in, EXTLA_PATH=../extla, never inferred from a sibling directory: tracking an unlocked branch is how extla v0.4.0's rename of TLA.* to ExTLA.* reached a pipeline unannounced.

State: running, live (armed timers the server would act on, 0..2), stale (cancelled-too-late ticks in flight, bounded by max_stale = 2), heat (0 = alpha below alpha_min; 1, 2 above), target_hot (alpha_target ≥ alpha_min), pending_start, and the history variable restart_started_stopped. Actions: deferred_start, start, stop(cancelled), restart, live_tick, stale_tick, manual_tick, set_alpha(h), set_alpha_target(t).

Invariants (checked over all 72 reachable states):

• running_iff_live_timer — running == (live == 1).
• at_most_one_live_timer — live <= 1.
• restart_never_starts — a :restart has never started a stopped simulation.

Liveness (cools_down, checked with the client quiet, env_active = false, under weak fairness on the tick and deferred-start actions): a target below alpha_min leads to not running.

Three constants select the intended behaviour (default true) or the defect the code shipped with, and test/specs/force_simulation_spec_test.exs proves the checker finds each when its switch is flipped: guarded_deferred_start (a :start_simulation after a :start must not arm a second timer), restart_keeps_stopped, and tagged_ticks (a tick carries its timer's reference, so a stale tick after a stop-then-start cannot start a second chain — a defect the review had not seen; the checker did). mix vis.verify runs mix extla.check on every model in @models and is the CI verify job; the code fixes are the second work item of proposal #5, after which §8.5's > Decision note is retired. ExTLA.GenServer conformance (Layer 3) is not attempted until a sibling project has proven the path.

6. Unit and regression tests

Implemented. mix test runs the ExUnit suite. test/visualize/defects_test.exs holds one regression per recorded defect fix (D-1, D-3, D-8 and the Data.range/3 step handling), and each future decision that changes behaviour MUST add a regression there or in the module's own test file.

The LiveView components have render goldens of the same kind as §1: test/visualize/components_test.exs renders every chart component of 10-liveview-integration §2 through Phoenix.LiveViewTest.render_component/2 on a small fixed dataset and on empty data, and asserts byte equality with test/support/components/<name>.svg and <name>_empty.svg (D-45). They are regenerated with mix run -e 'Visualize.ComponentsTest.write!()' only when a component's output changes on purpose, and the regeneration is committed with the change. test/visualize/components/tree_test.exs holds the hierarchy components of §3 the same way (mix run -e 'Visualize.Components.TreeTest.write!()'), and holds every d attribute of every golden under test/support/components/ to the form Visualize.IR.Path.to_string/1 prints — no space inside a d — so a component cannot reacquire a serialiser of its own (D-71).

The gallery has raster goldens (#474). test/visualize/render/raster_golden_test.exs rasterises every design of Visualize.Designs.names/0 — the thirteen the library's tests render — with Visualize.Render.to_png/2, from Visualize.Chart.generate(applied, root: true, resolve: :literal) at scale: 1 and background: "#ffffff", with the font configuration font_dirs: ["test/support/fonts"], system_fonts: false and generic_families: [sans_serif: "DejaVu Sans"], and compares each with test/support/raster/<name>.png:

• The bundled font. test/support/fonts/ holds one face, DejaVu Sans (DejaVuSans.ttf, DejaVu 2.37), under the Bitstream Vera licence with DejaVu's changes in the public domain, which permits redistribution with the notice; the notice is test/support/fonts/LICENSE. It is the only font the goldens see, so they do not depend on the host's fonts, and the CI image has none.
• No warnings. Each design returns {:ok, png, []}: resvg reports no family it could not resolve against the bundled font, and nothing else (09-rendering-backends §9.7). A design that gained a family the font configuration lacks fails here, not in a host's alert.
• Within a per-pixel tolerance. Both PNGs are decoded to RGBA pixels (Visualize.PNGPixels, test support: inflate the IDAT data and undo the five scanline filters of PNG §9). They MUST have the same width and height, and a pixel matches when each of its four channels is within 8 of the golden's; a design matches when at most 0.1 % of its pixels do not. The tolerance absorbs anti-aliasing that moves by a step between builds of the rasteriser, and a mark that moves, recolours or disappears exceeds it. The goldens are rendered by resvg CLI 0.48.1, the CI's pinned release (§4), and hold on 0.45.1, the supported minimum, with no pixel outside the tolerance (D-121).
• Drift. The set of files under test/support/raster/ MUST equal the set of design names, so a design added without a golden, or a golden left behind by a removed design, fails.
• Size. Thirteen 600 × 400 PNGs, about 130 KB in all; the font is 760 KB. Both are committed.

The goldens are regenerated with VISUALIZE_RESVG=/path/to/resvg MIX_ENV=test mix run -e 'Application.put_env(:visualize, :resvg, System.fetch_env!("VISUALIZE_RESVG")); ExUnit.start(autorun: false); Code.require_file("test/visualize/render/raster_golden_test.exs"); Visualize.Render.RasterGoldenTest.write!()' only when a design's picture changes on purpose — its SVG golden (test/support/designs_golden.txt) changes with it — or when a new resvg release moves pixels beyond the tolerance, with VISUALIZE_RESVG naming that release's binary (mix run does not read test/test_helper.exs, so the command sets the :resvg setting itself), and the regeneration is committed with the change. The regeneration command is in the test module's documentation, as for the other goldens.

The port is held to what it promises (#481, D-121), in test/visualize/render/raster_test.exs and, for the tests that change global state, test/visualize/render/raster_port_test.exs (async: false, so no other test runs beside them):

• No scheduler held. A peer node is started with one normal scheduler (:peer, +S 1:1, the suite's code paths). On it a ticker process loops on receive after 1, counting, while another process renders a design at a scale whose render takes far longer than a few dozen ticks. The test asserts a count: the ticker completed 20 ticks while the render was still in flight. A render that held the only scheduler — D-118's NIF — would let no tick run until it returned. No clock is read (§6, D-114).
• The timeout kills the processes. A render with timeout: 1 returns {:error, :timeout}, and so does Raster.run/4 over a stand-in program (/bin/sh -c 'sleep 60' through the same wrapper); for each, the test then signals the process group the os_pid led with signal 0 (kill -0 -- -<os_pid>), which fails once no process of the group exists.
• No temp file. With TMPDIR pointed at a fresh empty directory for the test (so System.tmp_dir!/0 and the child's environment both see it), a render leaves that directory empty and the working directory's listing unchanged.
• The binary and its version. A configured path naming nothing returns {:error, :no_rasterizer} even with a resvg on the PATH; a stand-in binary printing 0.44.9 is refused with {:error, {:rasterizer_version, "0.44.9", "0.45.0"}}, and one printing no version likewise.
• The warning text. A missing family is rendered through the real binary and read back as {:missing_family, list}, which pins Raster.warnings/1 to the CI's resvg; malformed SVG returns {:error, {:rasterizer, message}} with resvg's message.

test/visualize/theme_contrast_test.exs holds both built-in themes and every colour scheme to the contrast floors of 08-utilities §7.4.

test/support/contour_golden.txt records Visualize.Contour.compute/2, Visualize.Contour.render/2, Visualize.Contour.Density.compute/2 and Visualize.Contour.Density.render/2 over fixed grids and points, recorded from the implementation D-72 replaced, so the indexed grid and the linear tracing of 07-geo §6.3 are held to the rings the list-walking code traced; test/visualize/contour_golden_test.exs asserts byte equality and regenerates it (Visualize.ContourGoldenTest.write!/0) only when the tracing changes on purpose. test/visualize/contour_test.exs holds Visualize.Contour.Density.compute/2 on its default grid to a time bound and the cost of Visualize.Contour.compute/2 by a count, not a clock (#271): Visualize.Contour.cost/2 reports the work the tracing does — the cells read, the segments built and the hops the ring-following takes — and the test holds the hops to the segment count and the reads to the padded cells for 101² and 201² grids, which is D-72's claim stated as what it is; a wall-clock ratio, which the test asserted before, moved by a third between runs on a shared box and failed at 6.0–6.8× on a 6× bound eight times in one day without a defect. The density default is held the same way (#462): its reductions stay within a budget per padded cell and threshold, which is usable stated as work — the list-walking grid D-72 replaced spent hundreds of millions of calls on a 51 × 51 grid — and ten times the points costs under one and a half times the work, since the estimate's cost is the grid's and not the points times the grid; the time it takes is a :benchmark test.

No default-suite test asserts on wall-clock time; a timing is a :benchmark test (#462, #271, D-114). A cost claim — a scroll tick costs its strip and not its window, a curve is built in work linear in its points, a fixed-domain frame does not walk its rows — is a claim about the algorithm, and a wall-clock reading of it is a claim about the machine: a scheduler preemption during the cheaper of two timed runs flips a ratio, and an absolute bound moves with the box and with whatever else runs on it. Such a claim is asserted by the work done: what the code under test exposes as a count (Visualize.Contour.cost/2, the rows a strip carries, the path points a payload encodes), or the reductions the calling process spends, which Visualize.Work.reductions/1 (test/support/work.ex) reads around one warmed run of a function — the BEAM's own count of the work executed, unmoved by load. A measurement worth keeping is a test tagged @tag :benchmark (or a module tagged @moduletag :benchmark), which test/test_helper.exs excludes from the default run and mix test --only benchmark runs. test/visualize/no_wall_clock_test.exs enforces it: it scans test/**/*.exs and examples/test/**/*.exs for a clock read — :timer.tc, a monotonic, system or OS time, :os.timestamp, :erlang.statistics of wall clock or runtime — and fails on one that is not inside a test carrying @tag :benchmark in the attributes directly above it, nor in a module tagged @moduletag :benchmark. A helper that reads a clock outside any test fails it too: a benchmark times inline. The rule came from Visualize.Chart.CompiledTest's #404 scroll test, which compared two :timer.tc readings and failed under load (a 17 ms scroll against a 48 ms redraw on a one-third bound) while its algorithm was sound; #271 was the same defect in the contour test. The gallery under examples/ reads the same helper — examples/test/test_helper.exs requires test/support/work.ex — and holds every gallery chart's still render to a reduction budget by its declared cost class (spec/14 §12.6, #469).

The canvas hooks are executed, not only read (#458). test/visualize/hooks/executed_hooks_test.exs runs the bundle Visualize.Hooks.js_code/0 returns in node against a recording DOM stub, test/support/js/dom_stub.js, written for this and dependency-free: no npm, no jsdom, no build step. The stub gives a hook an element with dataset, querySelector('canvas'), appendChild and the attribute calls; a canvas whose getContext('2d') records every method call with its arguments and every property assignment, and whose width/height setters record the assignment and a clear, as setting either clears a canvas; an offscreen document.createElement('canvas') of the same kind; for BuilderHook, querySelector, querySelectorAll and closest over tag, class and attribute selectors, classList, contains and a getBoundingClientRect the scenario sets, and Stub.markup(tag, attributes, children), an element built from its HTML attribute names, so a scenario lays out the rows the builder rendered; pushEvent, which records, and handleEvent, which registers so a scenario delivers a payload; and setTimeout/clearTimeout/performance.now() on a manual clock the scenario advances. Visualize.JsHooks (test/support/js_hooks.ex) is the driver: it writes the bundle, the stub and one scenario to a temporary .mjs file, runs node on it with System.cmd/3, and decodes the recorded log the scenario prints as JSON with Jason; the assertions are ordinary ExUnit. The tests are tagged :js, and test/test_helper.exs excludes the tag, saying so in one line, when no node is on the path, so mix test never needs node; mix test --only js runs them, and is the verify:hooks job of §4. The first scenarios pin the defects that shipped green because nothing executed the hooks: a CanvasBinaryChart mounted at 600×400 whose data-width/data-height change to 1200×800 is resized on updated() and redraws its unchanged data-binary onto the cleared canvas (#456); a CanvasIncrementalChart resized the same way resizes its offscreen buffer and draws no scroll payload until a "full" one (#456); a stream of the current record types — path32, path_cubic32, cubic_run and their f64 forms, encoded by Visualize.Backend.CanvasBinary — replays without throwing and issues the moveTo, lineTo and bezierCurveTo it encodes (#270); a payload naming another data-frame draws nothing (#432); and payloads carrying a seq are acknowledged with pushEvent(ack, {seq, drawn, at}) at most once per data-ack-interval (D-107); and BuilderHook, over the stack panel's rows as the builder renders them with a group closed, sends the server's flat position for a drop below the closed group — the row's data-builder-layer, which Visualize.Chart.Builder.Stack.all/1 gives the row after it — and pushes no reorder for a group dropped directly below its own last row (#468). The sync group is executed too (#466, spec/10 §4.3). For it the stub also models document as an event target, with addEventListener, removeEventListener and dispatchEvent, each dispatch recorded, and a count of listeners per event name. It provides CustomEvent, document.body, createElementNS, and elements that take listeners, answer contains, closest and attribute selectors, and return a bounding rectangle the scenario sets. Three crosshairs of different widths in one group put their rules on the same domain x when one is hovered. A tooltip in the group shows the nearest datum. A chart in another group is untouched, a member whose domain does not hold the value draws nothing, a pointer leaving clears every member, and the publisher never handles its own echo. destroyed() leaves no listener, and a remounted member handles one event once. The shared brush is executed the same way (#467, spec/10 §6.4): the stub's elements also take insertBefore, and an <svg> answers createSVGPoint and getScreenCTM from its viewBox attribute and the bounding rectangle the scenario sets, as a browser maps a client point into user units. Brushing one of three charts of different widths draws the band on all three at the same domain extent and at each one's own pixels, clips it on a member whose domain holds part of it, shows nothing on one whose domain is disjoint, leaves another group untouched, and pushes exactly one brush_select, from the chart brushed, with the window's domain; a clear propagates with one brush_clear; and teardown and remount leave one listener. Removing sizeToElement() from either hook's draw path fails the resize scenarios. The crosshair is held to the drawing, not only to its arrays (#507, spec/10 §13.3): the hook tests had checked the hook against the arrays it was given and never the arrays against the chart, so a page that drew at one size and wrote its arrays at another shipped green. A container a few pixels taller than its <svg>, as an inline element's baseline leaves it, and one holding a CSS-scaled <canvas>, each put the marker on the vertex under the pointer within a pixel, in client pixels through the overlay's viewBox. The legend toggle is executed over real markup (#508, spec/10 §14.3). Visualize.JsHooks.markup/1 reads the SVG Visualize.Chart.render/2 returned and builds the same element tree out of Stub.markup calls, so the scenario runs over the chart as rendered and not over a hand-written imitation of it. For it the stub's selectors also take the descendant combinator, a space between compounds, which is how the hook writes .mark [data-series]. test/visualize/hooks/executed_legend_hook_test.exs renders a chart with a :line and a :circle mark, both with series, and mounts LegendHook over it. Clicking one entry gives display="none" to every element of that series: its path, each of its points and each of its labels. It gives it to no element of another series. A second click removes it from all of them, and updated() keeps the series hidden. The source-shape tests stay where they hold what execution does not: that the bundle defines and exports every hook once and the decoder precedes its callers (spec/10 §7), that the decoder has a case for every byte of Visualize.Backend.CanvasBinary.Opcodes and every sub-opcode — an exhaustiveness claim no finite stream proves — and the listener discipline of the pointer hooks (D-47), whose DOM the stub does not model.

test/visualize/documented_examples_test.exs holds every module the library ships whose moduledoc or function documentation carries an iex> example to being named by a doctest in some test file, without except: [:moduledoc] (D-90, D-93): a documented example that no test compiles is prose, and Visualize.IR.Element's had not been Elixir since it was written. The scan resolves each doctest through its file's aliases, so doctest Element beside alias Visualize.IR.Element counts. An example that cannot be deterministic — a random seed, a timestamp, a process identifier — is named in the guard's own exclusion list with the reason it is excluded, so a skipped example is data a reader can audit rather than an absence; the list is empty.

test/visualize/readme_example_test.exs runs the elixir block under the README's ## Example heading (#454): it reads README.md, takes the block by section rather than by position — the ## Installation block is a deps fragment and is not evaluable — evaluates it, and asserts what it promises: an SVG string from Visualize.Chart.render/1 and canvas commands from Visualize.Chart.render/2 with backend: :canvas. The README is the first code a reader runs, and until the guides' test below nothing else executed it: mix docs --warnings-as-errors checks its links, and doctests (D-90) cover documentation attributes, not a Markdown extra. Its example stopped compiling when #427 removed the size from cartesian/1 and stayed broken until #453; the test is what makes such a change a red pipeline. It is the same pattern as test/visualize/chart/example_test.exs, which reads 14-declarative-chart §13's worked example from the document itself. usage-rules.md carries no code blocks, so there is nothing of its own to run.

test/visualize/chart/spec_examples_test.exs does the same for every other elixir block of 14-declarative-chart (#460). It reads the document, pairs each block with the heading it sits under, and generates one test per block named by that heading, so a failure names the section and never a position. A block is evaluated with the context the section's prose assumes: import Visualize.Chart.Build, import Visualize.Chart, only: [var: 1] and alias Visualize.Chart.Use. A pipeline — a block whose value is Visualize.Chart.compose/1's, §16.2 and the second block of §16.3 — is asserted to return {:ok, design} and the design to validate (§10); a fragment — §3.6, the first block of §16.3, §16.4, §16.5, §19.3 and §19.6, which illustrate a shape rather than build a design — is asserted to evaluate. §13's two blocks are run by example_test.exs and are not run again; a heex block (§18.1) is not Elixir and is not read. Every other elixir block is run: the test fails on a block that is neither evaluated, nor §13's, nor marked, and holds the evaluated sections to an exact list, so a block added to the document is noticed rather than silently covered or silently skipped.

A block that is deliberately not runnable carries a marker on the line directly above its fence: <!-- not run: <reason> -->, an HTML comment that does not render, with a reason that is not empty. The test reads it, skips that block and that block only, and fails on a marker with no reason or with no elixir fence below it. A skip is therefore written in the document beside the code it excuses, where an editor of the section sees it, and a reader can audit every skip with one search. No block of spec/14 carries one today: all ten run.

Every guide's code is run (#518, #515). The guides under guides/ — getting_started.md, liveview.md, designing_charts.md and cheatsheet.cheatmd — ship in the package (files:) and are hexdocs extras in a Guides group placed after the README and before the usage rules and this specification, so a reader meets them first; ExDoc lists ungrouped extras before every group, so the usage rules and the changelog form a Reference group after it and the README alone is ungrouped. test/visualize/guides_test.exs runs every elixir block of every file matching guides/*.md or guides/*.cheatmd, and of README.md; the glob is read when the test compiles, so a new guide is covered with no edit to the test. There is one test per document. Its blocks run in document order in one evaluation environment — the binding and the imports and aliases an earlier block made — so a later block uses what an earlier one defined, as a reader typing them into IEx would. Each block is evaluated with the document as its file and its own first line as its line, so an error, a stack trace and a failure all name guides/<file>.md:<line>. A block MUST NOT warn: its diagnostics are collected and a warning fails the test. A line #=> <value> after an expression is the guide's claim about what that expression returns; the block is cut there, and the value evaluated so far MUST equal (==) the value the line evaluates to, so a guide cannot show a result the code no longer gives. getting_started.md, liveview.md and the README MUST each hold more than one elixir block, so a fence the reader stopped recognising fails rather than passing vacuously.

A block that cannot run outside a host carries a marker on the line directly above its fence: <!-- compile only: <reason> -->, an HTML comment that does not render, with a reason that is not empty — a LiveView needs an endpoint and a socket, a deps/0 fragment a mix.exs. It is the guides' counterpart of the not run: marker of spec/14 above, and a fence info string cannot carry it, since the Markdown parser ex_doc uses does not read a fence whose info string has a second word as a fence at all. A marked block is parsed with Code.string_to_quoted!/2; when every top-level form is a defmodule, it is also compiled, without warnings, and each module it defined is purged, so a guide's LiveView is held to the real names and arities of the library and of Phoenix. The test fails on a marker with no reason or with no elixir fence directly below it. The README's ## Installation block carries one; its ## Example block is run here as well as by the test above, which asserts more of it.