# DocShell usage rules

DocShell extracts documentation into versioned JSON and can project validated
collections into a portable site. It owns artifact and site contracts, route
validation, renderer admission, and static publication. It does not own a visual
renderer, host route taxonomy, authorization policy, tenancy, branding, or
deployment.

Requires Elixir 1.17 or later. Install `ash_oaskit`, `open_api_spex`, or `plug`
when the host uses the corresponding optional integration.

## Build and configuration

- Configure only the `:doc_shell` application. DocShell never reads another
  application's environment.
- Precedence is per-call options to `DocShell.Build.run/1`, then
  `config :doc_shell`, then package defaults. Defaults live in
  `DocShell.Config`, not in a config file, so a host that configures nothing
  still gets a valid artifact tree.
- Use `mix doc_shell.build` for host builds. It documents every module in the
  current application. The default task starts the app; pass `--no-start` to
  compile and load the app spec without starting the supervision tree. Call
  `DocShell.Build.run/1` directly, with an explicit `:modules` list, when you
  need a different set.
- Handle both `{:ok, result}` and `{:error, reason}`. Extraction stops at the
  first error and names the module or file at fault; it does not skip bad
  sources.
- Pass explicit `:modules`, `:guide_bases`, and `:livebook_base` values when the
  host layout differs from the defaults (`[]`, `["guides"]`, `"notebooks"`).
- Set `:collection` to a validated `DocShell.Generate.Collection` descriptor
  when the output will be imported into a documentation site. It records the
  caller-supplied source revision and tree digest; DocShell never invokes Git
  or fetches the descriptor URLs. The public output then includes the
  enveloped `collection.json` provenance artifact.
- Treat changelog/release notes as a source adapter. The default
  `DocShell.Generate.Changelog.Sources.MarkdownFile` reads `CHANGELOG.md`, but
  graph/database/CMS hosts should implement
  `DocShell.Generate.Changelog.Source` and pass `:changelog_options`. Use
  `DocShell.Generate.Changelog.from_markdown/2` when the dynamic source stores
  Markdown.
- Leave `:open_api_adapter` unset to emit a valid empty OpenAPI 3.1 document.
  This is a supported configuration, not a degraded one.
- Use `DocShell.Build.run/1`'s return value to feed a database or knowledge
  graph, with `write: false` when the files are not wanted. The return value is
  richer than what is written: entries keep their parsed `ast` and nothing is
  filtered out. Projector backlinks remain in memory and have no disk artifact.
- Set `:presentation_source` to a `DocShell.Presentation.GraphProjector`
  implementation to have the build use a host projector. `:path_builder`,
  `:skip_empty`, `:search_tokens`, and `:search_members` pass through to the producer.
  Member search defaults off. Enable it to append module member names/arities,
  signatures and parsed docs to page search text, without changing content or
  overriding `skip_empty`; malformed member Markdown fails with the module ID.
- Set `:openapi_spec_path` when external tooling needs a bare OpenAPI file.
  Put it outside the artifact directories — `DocShell.Web.Cache` rejects a
  directory holding an unenveloped `.json`.

### Default API identity

The default OpenAPI 3.1 document includes `info.title` and the configured
`api_version` as `info.version` (default `"0.1.0"`).

### Configuration errors

`Build.run/1` rejects malformed options and unknown per-call keys before
extraction. Unknown application environment keys remain ignored. Invalid guide
identities, titles, audience, and locale return errors naming the field and file.
Guide IDs and titles accept nonempty strings or numeric/boolean scalars.

### Output destinations

Public and private output directories must be disjoint. The optional raw
OpenAPI destination must lie outside both. Conflicting paths fail before
extraction or writes; use dedicated directories without symlink aliases.

### Failed builds and recovery

Builds stage all JSON and back up existing files before publishing. Use the transaction's
`delete: [path]` option for removals so locks and rollback cover them; duplicate
targets, directories and symlinks are rejected before publication. Returned
publication failures restore earlier files, including deleted artifacts; rollback failures report retained
backup paths. Cooperating builds use `.doc-shell-build.lock` directories. After
a process or machine crash, recover retained backups and remove stale locks
before rebuilding. Files still publish individually, so cache reloads validate
generation IDs and keep the last complete snapshot. Use dedicated output
directories without external writers or symlink aliases.

## Artifact contract

### Collection implementation guidance

Follow the collection integrity requirements in
`docs/specs/DSH.01-documentation-sites.md` and the acceptance status in
`docs/plans/documentation-sites.md`. Source paths identify files; document IDs
identify records, so separate changelog releases may share one source file.
Include synthetic OpenAPI identity in duplicate checks. Require complete
provenance and a build/load round trip. Reject incompatible custom collection
projections before publication; never reintroduce filtered public bodies implicitly.
Unknown source indexes referenced by provenance use the generic v1 entry shape.
Loading rejects missing or repeated provenance, duplicate document IDs and
malformed source indexes. The ID `openapi` is reserved in collection mode.
Custom projectors must preserve extracted ASTs and exact JSON numeric types (omitted empty ASTs reconstruct
as `[]`). Dynamic source locators belong in metadata such as `source_ref`;
an embedded index AST must agree with content, and OpenAPI owns no page AST.
collection `source_path` values must be relative filesystem paths.
Canonical digesting rejects duplicate encoded keys, and malformed boundary input
returns tagged errors. Keep deletion inside publication locks and rollback.
Artifact writers validate protocol-generated JSON and return host encoder errors
without replacing files. Changelog extension fields must remain native JSON in
both collection and ordinary builds, including `write: false`.
ExDoc member metadata is normalized with duplicate-key checking. Keep custom
`@doc` metadata keys distinct after conversion to strings.
Import assumes a stable caller-owned directory and uses finite resource limits;
hashes and path checks are not source authentication or an OS sandbox.

Use `Collection.load/2` or `load_many/2` with positive keyword overrides for
`max_file_bytes` (32 MiB), `max_total_bytes` (2 GiB including the manifest),
`max_artifacts` (256), `max_sources` (100,000), `max_json_depth` (64), and
`max_collections` (256). Except collection count, these budgets are per corpus.
No unlimited value is supported. Root symlinks and `..` traversal fail, including
symlinks spelled with a trailing slash or `/.`. Ancestors of the existing working
and system temporary directories are trusted aliases; other symlink components
are rejected. Duplicate JSON object keys also fail before digest validation.

- Treat `DocShell.schema_version/0` and the `doc-shell/v1` shapes as public API.
  Never invent fields or change a field's type in place.
- Treat the v1 source catalogue as additive. A manifest may list a new
  per-source artifact and `kind` is an open string; generic consumers ignore
  unknown files and kinds, while selective consumers may read only their
  allow-listed artifacts. Removing an existing file or changing an existing
  field's name or type requires a schema-version change.
- A site-ready collection uses the additive `doc-shell-collection/v1` payload
  in `collection.json`. Its portable descriptor excludes the local artifact
  directory and includes source identities, source provenance, artifact
  digests, and a canonical aggregate digest. Load it with
  `DocShell.Generate.Collection.load/1`; the loader rejects missing or
  unlisted files, symlinks, path escapes, mixed generations, descriptor
  differences, and digest mismatches before qualifying IDs as
  `collection_id:document_id`.
- Read the version from `DocShell.schema_version/0` rather than writing the
  literal `"doc-shell/v1"`.
- Read and write artifacts through `DocShell.Artifact`. Do not bypass the
  envelope or encode artifact JSON by hand.
- Treat `generation_id` as an opaque snapshot identity. Every artifact and the
  manifest from one build must carry the same value; never synthesize or reuse
  one across builds.
- Keep generated content renderer-neutral: no host UI, routing, tenant, or
  authorization assumptions inside an artifact.
- Produce presentation data with `DocShell.Presentation.NavigationItem`,
  `SearchEntry`, and `Backlink` structs, not bare maps.
- Validate graph-backed output through
  `DocShell.Presentation.GraphProjector.project/2` before exposing it.
- Resolve repository membership, exact revisions, graph position and disclosure
  policy in the host before invoking the projector. DocShell consumes one
  admitted presentation; it does not discover repositories or filter an
  internal graph into a public site.
- When API reference accompanies a hosted graph presentation, the host must
  validate that both inputs came from the same admitted source view, graph
  position, locale, release, and disclosure surface before rendering. Never
  reuse a public artifact for an internal projection or attach a process-global
  current spec to an older presentation. DocShell does not yet validate this
  pair; DSH-P10 plans a common opaque publication binding and pre-render check.
- Do not manufacture per-package or per-repository collections merely to feed a
  hosted graph projection. Collections/cohorts are the portable site path, not
  the hosted authorization boundary.
- Changing a `doc-shell/v1` shape is a breaking change to every producer and
  renderer at once. Adding an optional field is usually safe; renaming,
  removing, or retyping one is not.

### Rendering untrusted content

The AST preserves raw HTML and URL schemes. Renderers must allow-list tags and
attributes, reject unsafe URL schemes, and escape text for their output context.
Parsing Markdown is not sanitization.

### Document identities

Document IDs must be nonempty and unique across all sources, including entries
filtered from presentation. Duplicate IDs return an error naming both sources.
Overlapping guide directories extract each normalized path once.

### Recursive content validation

Host projectors and changelog sources must provide complete recursive AST nodes
whose extension fields are native JSON. Presentation IDs must be nonempty and
unique within navigation (including descendants) and search; shared IDs use the
same path. Local leaves/search results require content. Groups with children and
absolute HTTP(S) links may omit content; content may be hidden from both indexes.
Backlink targets require content, while origins may belong to the wider host graph
provided they do not contradict a known path. Default ordering is kind/title/ID.

Metadata keys must be strings. Invalid nested content fails validation before
output is written. `DocShell.Ast.valid?/1` checks node lists.

### Legacy envelope compatibility

`Artifact.read/1` and `read_envelope/1` accept legacy v1 envelopes without
`generation_id`. A present ID must be a nonempty string. Runtime caches require
an ID on every artifact and manifest to verify that they form one generation.

### Concurrent artifact writers

Individual artifact writes use exclusively created random temporary files in
the destination directory, so independent BEAM instances cannot share a
temporary file. A rename publishes each complete file.

### Search text

Search content preserves adjacent inline text, including words split by
formatting. Block elements and line breaks add separators; image alt text is
searchable. Token generation uses this same text.

### Document paths

Each document path is calculated once and reused by navigation and search.
Default paths percent-encode kind and ID as individual URL segments. Use a
custom `path_builder` when IDs intentionally represent a path hierarchy.

## Portable site projection

- Load every input with `DocShell.Generate.Collection.load/1` before projection.
  `DocShell.Generate.Cohort` hashes portable descriptors, content digests, and
  the selected profile; local paths and generation IDs never enter that digest.
- Implement `DocShell.Presentation.SiteSource` to choose page declarations,
  routes, navigation, redirects, locales, and site metadata. The callback is a
  policy seam, not an extractor or renderer: it receives loaded collections and
  returns inert JSON-compatible values.
- Put provider-specific file links in a page declaration's `source_url` and
  `edit_url`. Without an override, `source_url` is the collection descriptor's
  exact URL and `edit_url` appends the encoded source path to `edit_base_url`.
  DocShell validates these HTTP(S) links but never invents forge-specific URL
  segments.
- Use `DocShell.Presentation.SiteSource.Default` for deterministic flat routes.
  Do not treat its collection/kind/document layout as product taxonomy.
- Call `DocShell.Presentation.SiteProjector.project/1` to join declarations to
  exact documents. It derives anchors, resolved local links, breadcrumbs,
  reading flow, search records, content digests, and renderer requirements.
- Pass an origin such as `https://docs.example.com` as `:canonical_origin`.
  Origins with a path, query, or fragment are rejected; put the mount path in
  the site declaration's `base_path` instead.
- Navigation groups are structural and have no route. Projected breadcrumbs use
  `DocShell.Presentation.Breadcrumb`; group items have `path: nil`, while the
  page item carries its canonical route.
- The `"public"` profile excludes page declarations or derived page status set
  to `"draft"` or `"private"`. A private corpus is never inherited from a
  public descriptor; pass it as a separate loaded collection under host policy.
- Keep presentation limits finite through `DocShell.Presentation.Limits`.
  Projection checks collections, pages, AST bytes/depth, identities, routes,
  navigation, and redirects before returning a `Site`.

## Renderers and static publication

- Implement `DocShell.Presentation.Renderer` in a renderer package or host. A
  renderer receives only `Site`, `Page`, and inert `Renderer.Context` values; it
  does not choose routes, authorize users, fetch sources, or write output files.
- Return logical local `Asset` values and a validated `Renderer.Capabilities`
  declaration. Build tools are not served-runtime requirements. Static output
  may use local browser JavaScript but cannot require live transport.
- Use `Renderer.admit/3` for hosted output. `StaticExporter.export/1` performs
  the same admission automatically and rejects unsupported essential features.
  Optional degradation requires the exact projected fallback digest.
- The built-in `SearchAdapter.JSON` emits a deterministic local index and offers
  `query/3` as the portable substring/filter reference. Custom adapters consume
  the same `SiteSearchEntry` records and return local assets plus inert query
  metadata.
- `StaticExporter.export/1` owns final paths, content hashes, local-link checks,
  output budgets, staging, replacement, and rollback. Renderer and search
  callbacks return data; they never receive the destination.
- Supply only a stable caller-owned destination. Existing non-directory or
  symlink destinations fail. A lock directory serializes cooperating exports;
  failed rendering or validation leaves the prior tree untouched.
- `site-manifest.json` records every generated payload except itself, avoiding a
  recursive digest. `generated_at` is absent unless the caller supplies it.
- Use `DocShell.Presentation.Conformance` and the packaged
  `priv/contracts/site-conformance-v1.json` fixture to compare normalized hosted
  and static semantics. Do not compare framework-specific HTML bytes.

## Source integrations

- Implement `c:DocShell.Generate.OpenApi.Adapter.load/1` for a new OpenAPI
  source. Return `{:ok, map}` with an `openapi` key of `"3.0.x"`, `"3.1.x"`, or `"3.2.x"`,
  or `{:error, reason}` with a reason worth reading in a failed build.
- Implement `c:DocShell.Generate.Changelog.Source.load/1` for a new
  release-note source. Return validated DocShell changelog entries, or fetch
  Markdown dynamically and pass it through
  `DocShell.Generate.Changelog.from_markdown/2`.
- The bundled Markdown parser accepts git_ops headings and Keep a Changelog
  headings, including parenthesised or hyphenated dates, inline or
  reference-style links, unlinked releases, and SemVer prereleases.
- Resolve optional libraries at runtime with `Code.ensure_loaded?/1`. A
  compile-time reference breaks every host that does not install the library.
- Preserve Markdown as the renderer-neutral AST from `DocShell.Ast`. Never emit
  HTML from an extractor.
- Surface malformed configured sources as errors; do not silently discard them.
- Normalize input metadata with `DocShell.Json.normalize/1` and handle
  converted-key collisions as errors. Structs become their `String.Chars`
  text where they have one and their `inspect/1` form otherwise.
- Read `meta["moduledoc"]` (`"present"`, `"hidden"`, `"none"`) rather than
  inferring documentation coverage from an empty `ast`.

### Guide line endings

YAML frontmatter accepts LF, CRLF, and CR line endings, including a closing
`---` delimiter at end of file.

### Markdown titles

Guide and notebook titles come from the first top-level parsed H1, including
Setext headings. Inline formatting is flattened, and headings inside code
examples are ignored. Explicit guide frontmatter titles still take precedence.
Explicit titles avoid a title-only parse. Unfenced bodies reuse their AST;
fenced input retains defensive title parsing so malformed closings cannot expose
code comments as titles.

### Changelog source validation

Every changelog source entry must be a valid entry map; `nil` and other invalid
entries return `{:error, {:invalid_changelog_entry, entry}}`.

### Changelog Markdown context

Changelogs are parsed as complete Markdown documents before splitting on
top-level release headings. Code examples remain within their release, and
reference links resolve across the whole document. Parse errors in any part
of the source return a source-tagged error.

### JSON metadata normalization

Use `DocShell.Json.Canonical.encode/1` or `digest/1` at serialization boundaries;
both return tagged errors and reject duplicate encoded keys. `Collection.digest/1`
requires encodable input and raises `ArgumentError` otherwise. Invalid UTF-8
source files fail with `:invalid_utf8`, tagged with the file by extraction/build.

Metadata preserves JSON scalars and uses UTF-8 string keys. Unsupported terms
become inspected text; improper list tails become a final array value.
`DocShell.Json.normalize/1` rejects converted-key collisions. The legacy
`stringify/1` keeps string keys when a collision occurs. Guides use `normalize/1`.

### OpenAPI version support

Raw and custom adapters accept OpenAPI 3.0, 3.1, and 3.2 documents without
rewriting their fields. Validation checks the version and unambiguous JSON encoding; source
libraries own schema validation. The default document remains OpenAPI 3.1.

### Optional integration dependencies

Plug is an optional package dependency because web modules compile against it.
AshOaskit is a development/test fixture; hosts using its runtime adapter install
AshOaskit themselves. Core consumers do not resolve its dependency tree.

## Optional web serving

- Add `DocShell.Web.Cache` to a supervision tree before serving artifacts, and
  call `DocShell.Web.Cache.reload/1` after a rebuild.
- Keep `manifest.json` beside the artifacts it lists. Cache startup and reload
  reject missing manifests, unlisted files, and mixed generation identifiers.
- Use `DocShell.Web.Plug` only when Plug is installed. The Plug, controller, and response modules
  compile conditionally; the cache can be used without Plug.
- Supply host authorization through the plug's `:gate` option — a unary function
  or an MFA tuple, returning `:ok` or `true` to allow. Omitting it serves
  everything to everyone.
- For private documentation, base the gate on server-validated request state and
  keep public/private routes and caches separate unless complete authorization
  partitioning is proven. Client-side redirects, audience parameters and raw
  browser storage are not gates.
- Use `DocShell.Web.Controller.show/2` instead when the host wants its own
  pipeline in front; put authorization in a plug there.
- DocShell must not implement application-specific access policy.

### Supervising named caches

A cache child specification uses its registered name as its child ID. Multiple
named caches can be listed directly in one supervision tree.

### Cache ownership

Cache ETS tables permit direct concurrent reads, but only the cache process
may write them. Individual fetches are consistent; separate fetches may cross a
reload. Use `Cache.snapshot/2` for a caller-owned copy of all envelopes from one
generation, including the manifest. It returns `{:ok, %{generation_id: id,
artifacts: envelopes}}` and remains valid after reloads, at the cost of retaining
that copy in caller memory. Snapshot calls queue behind reloads.

`Cache.reload(server, timeout)` and `snapshot(server, timeout)` default to 5,000
milliseconds. They follow `GenServer.call/3`: timeouts/unavailable processes exit
the caller, and timing out does not cancel a queued or running reload.
Reusing an active generation ID with changed envelopes returns
`{:error, {:generation_id_reused, id}}` and preserves the prior snapshot.

### HTTP response caching

HTTP serving caches encoded JSON and an ETag per generation. GET and HEAD
share headers; matching `If-None-Match` requests return 304 after authorization.
Other methods return 405 with `Allow: GET, HEAD`. The host retains control of
Cache-Control and Vary. Encoding happens during cache publication, not requests.

## Further reading

- [Build pipeline](https://hexdocs.pm/doc_shell/build-pipeline.html)
- [Artifact contract](https://hexdocs.pm/doc_shell/artifact-contract.html)
- [OpenAPI adapters](https://hexdocs.pm/doc_shell/openapi-adapters.html)
- [Serving artifacts](https://hexdocs.pm/doc_shell/serving-artifacts.html)