Portable documentation sites implementation plan

Copy Markdown View Source

This plan implements DSH.01 in dependency order. Each work package requires executable evidence before it is complete. Later work must not weaken an accepted error, limit, or compatibility rule to make its fixtures pass.

PackageStatusRequiresDeliverableAcceptance
DSH-P01completeExisting doc-shell/v2 readerAdditive collection.json, DocShell.Generate.Collection and collection descriptorDSH-V01/V02 plus collection integrity requirements below; no network or process startup
DSH-P02completeDSH-P01DocShell.Generate.Cohort, canonical payload digest, normalized source provenanceDSH-V03; digest excludes time, random IDs, and absolute workspace paths
DSH-P03completeDSH-P01/P02Site, Page, navigation/link types and SiteSource validationDSH-V04–V06; all public structs/functions documented and typed
DSH-P04completeDSH-P03Heading/anchor, reading-order, locale/audience/version, canonical/source/edit and redirect projectionDSH-V04–V06 with property tests for route and anchor normalization
DSH-P05completeDSH-P03/P04Section-aware search records, built-in deterministic JSON adapter and adapter behaviourDSH-V07; fixed query corpus and browser-neutral output
DSH-P06completeDSH-P03–P05Renderer behaviour, normalized capability/requirement/context/asset contracts, reserved feature IDs and shared fallback/enhanced/connected conformance fixturesDSH-V09/V10; fixtures published with the package for independent renderers
DSH-P07completeDSH-P02–P06Atomic static exporter, site manifest, sitemap, robots and llms outputsDSH-V08; clean build, rollback, subpath and output-bound evidence
DSH-P08in progressDSH-P06/P07Hosted/static semantic parity and live-capability degradation toolCore comparison is complete; DSH-V11 still requires two independent renderer implementations
DSH-P09completeDSH-P01–P08Documentation, Livebooks, benchmarks, compatibility and packaged-consumer qualificationDSH-V12 and the core release evidence recorded below
DSH-P10plannedDSH-P03/P06Preserve the hosted graph/source-cohort boundary and validate a host publication/OpenAPI pair before renderer invocationDSH-V13; hosted projection needs no synthetic per-repository collection; cross-surface and mixed-generation pairs fail
DSH-P11in progressDSH-P03/P06/P07Freeze portable artifact block/profile/receipt schemas and v2 state-specific capability fixturesStatic records and tests implemented; complete schema/limit and compatibility matrix remains open
DSH-P12in progressDSH-P11Add explicit per-call profile adapters, block validation and requirement/admission projectionChart/drawing fixtures, runtime selection, callback containment, concurrent isolation and three-seed model checks pass
DSH-P13in progressDSH-P12/P07Join verified local variants, search/text fallback and atomic exportSelected closure, original-byte checks, prior-tree preservation, emitted-script refusal and root/subpath file export tests pass
DSH-P14in progressDSH-P13; actual producer and independent readersQualify document previews, renderer/native semantic parity and fresh package consumersFirst producer's installed archive/static document join and full package gate pass; native/browser/manual target evidence remains open

P10 is outside the core site qualification record below. The existing generation_id joins artifacts written by one DocShell.Build run; it does not bind independently supplied hosted graph and OpenAPI inputs. The build invokes the OpenAPI adapter and presentation source separately, then creates that ID when writing artifacts.

Before implementing validation:

  • Define an additive pair contract and its admission point. Preserve existing doc-shell/v2 fields and keep host authorization policy outside DocShell.
  • Require comparable context on both sides. An opaque binding alone cannot detect a token reused across source views, graph positions, locales, releases, or disclosure surfaces.
  • Keep claim authentication in the host. DocShell checks declared equality before renderer invocation.

Public API shape

Foundation hardening before site work

This work qualifies the P01 collection implementation. It repairs the extraction library and does not advance P02–P09 or establish renderer/export conformance.

WorkStatusRequired evidence
Clarify source/file identities, projection compatibility and canonical JSONcompleteSpecification, README, usage rules and notebook guidance agree
Check JSON, descriptor and source input boundariescompleteCanonical JSON fixtures/properties, malformed descriptor/extraction regressions, consistent OpenAPI errors and invalid UTF-8 build tests
Validate complete indexed provenance and preserve source extensionscompleteCollection integrity regressions and generated build/load properties cover the real multi-release corpus, reserved/duplicate IDs, malformed/future artifacts and projector failure before publication
Secure and bound corpus loadingcompleteRoot spellings and parent-symlink regressions; exact file/total/count/depth limits; JSON escape properties and duplicate-key rejection; prior compatible corpora
Make stale deletion transactionalcompleteTransaction tests cover deletion rollback and lock ownership; collection regression verifies unchanged manifest on obstructed deletion
Strengthen presentation and cache contractscompleteEqual-title regression/properties; semantic reference and JSON-extension checks; opt-in member search; concurrent snapshots and configurable reload timeout tests
Qualify performance and package consumerscompleteCollection scaling and full benchmark suite; nine locked/unlocked/compatible-minimum consumers; four notebooks; complete mix check; repeated malformed-input and publication review

Keep the existing public facade when separating descriptor, provenance, digest, and filesystem responsibilities. Commit each coherent behavior with its tests. Mark a row complete only with executable evidence and update docs in that commit.

Qualification record — 2026-09-11

Local verification used Elixir 1.18.4 and OTP 27.3.4.15. The existing CI matrix remains unchanged; other Elixir/OTP pairs were not locally rerun or remotely dispatched as part of this work.

CheckResult
Full mix check, without automatic retry-only behavior277 tests, 28 properties, 16 doctests; zero failures; 97.4% coverage
Compilation, strict Credo, Dialyzer, Doctor, ExDoc, formatting and dependency auditsPass; 100% documentation/typespec coverage; no ignored Dialyzer warnings added
Compile without optional integrationsPass
Executable notebooksAll four pass, 55 executable cells
Unpacked Hex archive consumersCore, Plug and Ash pass for locked, unlocked and selected compatible-minimum dependencies
BenchmarksAll suites run; measurements refreshed, including preparation and import at 1,000–16,000 documents

The minimum consumer set is Jason 1.4.0, EarmarkParser 1.4.8 and YamlElixir 2.11.0, except the Ash consumer needs Jason 1.4.5 for its Decimal 3 dependency. Old EarmarkParser versions emit upstream deprecation warnings on modern Elixir; these are not suppressed. This is selected direct-dependency qualification, not an exhaustive historical/transitive dependency matrix.

On the original synthetic 16,000-document diagnostic fixture, median-of-three preparation time changed from 2,766.9 ms to 48.9 ms and import from 5,454.7 ms to 876.3 ms. These local observations are not timing assertions or asymptotic proofs. The committed collection benchmark records independent elapsed-time and allocated memory measurements. Source loops now use indexed lookup and prepend/reverse; canonical object ordering still requires key sorting.

Follow-up review also closed integer/float projection mismatches, contradictory embedded ASTs, orphan OpenAPI content, protocol-encoder failures, colliding member metadata keys and changed bytes under a reused cache generation ID. Descriptor, filesystem, provenance and canonical JSON responsibilities have separate modules; the compatible collection facade remains.

Import requires a stable caller-owned directory. Checksums establish internal consistency, not source authenticity, and publication is not power-loss atomic. The package provides neither an OS sandbox nor a visual renderer. No workflow, remote ref, release tag, repository visibility or dependency lock was changed. Do not change schemas, dependency ranges, workflow checks, release refs, or repository visibility merely to make a check pass.

Core site qualification record — 2026-09-22

Core site publication was qualified locally on Elixir 1.18.4 and OTP 27.3.4.15. Fresh endpoint builds also passed on Elixir 1.17.3/OTP 27.3.4.15 and Elixir 1.20.4/OTP 28.5. These checks qualify the DocShell package; they do not substitute for the two independent renderer results still required by P08.

CheckResult
Full mix check, without retry-only behavior303 tests, 28 properties, 16 doctests; zero failures; 95.3% coverage
Compilation, strict Credo, Dialyzer, Doctor, ExDoc, formatting and dependency auditsPass; 100% documentation, moduledoc and typespec coverage
Compile without optional integrationsPass
Supported-version endpointsFull tests pass on Elixir 1.17/OTP 27 and Elixir 1.20/OTP 28
Executable notebooksAll five pass, 62 executable cells
Unpacked Hex archive consumersCore, Plug and Ash pass with locked, newly resolved and selected compatible-minimum dependencies
Package contentsSite contracts, conformance fixture, specification, plan, notebooks and benchmark reports are present
Site benchmarkProjection and complete staged export measured at 10, 100 and 500 pages, with search/output sizes and allocated memory

The 500-page local benchmark recorded a 204.50 ms median for projection and a 338.92 ms median for full static replacement. Benchee reported 585.74 MB and 311.04 MB of allocated memory respectively; these are allocation measurements, not peak resident memory or performance assertions.

The minimum consumer set retains the dependency versions recorded in the earlier qualification section.

Old minimum EarmarkParser releases emit upstream deprecation warnings on modern Elixir, and Ash emits optional Igniter warnings when Igniter is absent; neither warning originates in DocShell.

Site modules

The implementation exposes these public modules while keeping projection and filesystem publication mechanisms in private supporting modules:

DocShell.Generate.Collection
DocShell.Generate.Cohort
DocShell.Presentation.Limits
DocShell.Presentation.Site
DocShell.Presentation.Page
DocShell.Presentation.Link
DocShell.Presentation.Heading
DocShell.Presentation.SiteSource
DocShell.Presentation.SiteProjector
DocShell.Presentation.SearchAdapter
DocShell.Presentation.Renderer
DocShell.Presentation.Renderer.Capabilities
DocShell.Presentation.Renderer.Capability
DocShell.Presentation.Renderer.CapabilityRequirement
DocShell.Presentation.Asset
DocShell.Presentation.StaticExporter
DocShell.Presentation.Conformance

Expected failures use tagged tuples and stable reason atoms with structured context. Host callbacks are invoked behind exception capture and result-shape validation, following DocShell.Presentation.GraphProjector.

Compatibility sequence

  1. Add collection and site contracts without changing existing doc-shell/v2 files.
  2. Generate doc-shell-site/v1 only when a caller invokes the site builder.
  3. Publish conformance fixtures before renderer packages claim support.
  4. Qualify the existing Svelte renderer and one independent HTML renderer.
  5. Treat any required change to an existing doc-shell/v2 field as a separate schema decision with a migration fixture.
  6. doc-shell/v2 is such a decision: module navigation items carry member references instead of member records, members carry doc_state, member metadata defaults to an allow-list, and files are compact canonical JSON. test/fixtures/presentation-v2.json is its fixture; v1 files are rejected.

Required quality evidence

Portable artifact delivery

The experimental extension supplies generic document capabilities. P11–P14 are in progress: the static implementation and its tests exist, while the complete compatibility and reader qualification matrix remains open. The new API and schema IDs apply to this implementation; they do not alter v1 meaning or establish independent reader qualification.

The October commit review retained regressions for three implementation defects: runtime-list order changing admission, script-free exports accepting JavaScript MIME parameters/aliases, and empty sites skipping artifact-option validation. All three first failed on their observable behavior and then passed after corrections in the admission/export owners. The focused portable artifact suites pass 18 tests. These results do not close the remaining schema/limit, independent reader, native, or human target qualifications.

P11 fixes original block bytes, descriptor/schema digests, semantic subjects, all variant bytes and expected refusal/admission outcomes. Freeze a technical illustration and an unrelated chart fixture before writing core adapters. Include mixed static/JS/live variants, no-JS policy, duplicate descriptors, unknown optional/essential profiles and altered block/fallback bindings.

P12 extends the existing AST and site projection owners; it adds no global plugin registry, remote discovery, producer dependency or rendering process. Compare admission with an independent small model at three fixed seeds and exercise two differently configured callers concurrently. Every callback fault returns a located error rather than partial publication.

P13 passes prepared local bytes through the existing asset/export owner. A script-free variant must survive actual offline/JS-disabled presentation; declaring runtime: [] is not a passing result. Asset source disclosure also applies to thumbnails, descriptions, search and fallbacks. Corrupt or absent required assets preserve the previous complete output tree.

P14 supplies a local review page from the exact projected data, including artifact/variant IDs, anchors, selected capability, limitations, fallback and generated SVG/PNG.

The artifact producer supplies reference/candidate/difference images; DocShell does not calculate geometry or approve baselines.

Test / and subpath exports, print, dark/monochrome/high zoom and reader restart. An HTML reader and a native artifact consumer record the same semantic observations; native paint and human assistive tasks require their own target evidence.

Run the full package gate and current CI compatibility matrix when implementing these rows. Ship the new fixture/schema documents in the fresh package archive. The current v1 renderer pair and hosted publication/OpenAPI obligations remain open independently; artifact previews cannot close them.

Existing site quality evidence

  • Unit tests cover every structured error and public default.
  • StreamData covers route normalization, anchor uniqueness, input ordering, canonical JSON, and output path containment.
  • Corpus fixtures include modules, guides, Livebooks, release entries, OpenAPI, unknown source kinds, nested navigation, multiple locales, and versions.
  • Hosted graph fixtures prove DocShell consumes one already-authorized projection without repository discovery, cohort inference or renderer-side privacy filtering.
  • Hosted public/internal fixtures carry matching narrative/OpenAPI publication, graph-position, locale and release bindings; cross-surface and mixed-generation pairs fail before renderer invocation.
  • Static fixtures run from / and a repository subpath with JavaScript blocked and enabled.
  • The package archive contains the public schemas, conformance fixtures, and usage documentation.
  • Benchmarks record projection and export time, allocated memory, page count, search size, and output bytes for small and large fixed corpora.