The Build Pipeline

DocShell.Build.run/1 reads modules, Markdown guides, Livebook notebooks, release notes, and an OpenAPI document, then projects them into JSON artifacts for a renderer.

This tutorial follows one complete build from source files to artifacts. The examples create a disposable documentation set under your system temp directory, so running the notebook does not touch priv/doc_shell/ or depend on files in this repository.

What you will learn

• How to prepare guide and notebook sources for a build
• What DocShell.Build.run/1 returns in memory
• Which files are written to disk
• Where filtering, paths, search text, and OpenAPI fit in the pipeline
• How DocShell fails when a configured source is invalid

Setup

Run this notebook from Livebook's default standalone runtime. The setup cell installs the matching DocShell release from Hex. Release tooling updates this version together with the package and the notebook links.

Mix.install([
  {:doc_shell, "== 1.0.0"}
])

Application.ensure_all_started(:doc_shell)

Create a small documentation set

This tutorial creates a small documentation set:

• one Markdown guide with YAML frontmatter
• one .livemd notebook
• one real module from DocShell itself, DocShell.Config

workspace =
  Path.join(
    System.tmp_dir!(),
    "doc_shell_build_pipeline_#{System.unique_integer([:positive])}"
  )

File.rm_rf!(workspace)

guide_dir = Path.join(workspace, "guides")
livebook_dir = Path.join(workspace, "notebooks")
public_dir = Path.join(workspace, "public")
private_dir = Path.join(workspace, "private")

Enum.each([guide_dir, livebook_dir, public_dir, private_dir], &File.mkdir_p!/1)

guide_path = Path.join(guide_dir, "getting-started.md")

File.write!(guide_path, """
---
id: getting-started
title: Getting Started
audience: developers
locale: en
---

# Getting Started

Start with a small documentation set and add sources deliberately.
""")

livebook_path = Path.join(livebook_dir, "incident-playbook.livemd")

File.write!(livebook_path, """
# Incident Playbook

This notebook is documentation too. DocShell indexes the source; it does not
evaluate the code cell.

```elixir
System.system_time(:second)
```
""")

%{
  workspace: workspace,
  guide_path: guide_path,
  livebook_path: livebook_path,
  public_dir: public_dir,
  private_dir: private_dir
}

Run the build

DocShell.Build.run/1 accepts per-call options. These override host configuration, which overrides DocShell defaults. This example uses explicit options so it runs independently of host configuration.

build_opts = [
  modules: [DocShell.Config],
  guide_bases: [guide_dir],
  changelog_source: nil,
  livebook_base: livebook_dir,
  public_dir: public_dir,
  private_dir: private_dir
]

{:ok, result} = DocShell.Build.run(build_opts)

Map.keys(result) |> Enum.sort()

The return value always has the same top-level keys:

• :modules — module documentation extracted from the BEAM docs chunk
• :guides — Markdown guides with parsed body and metadata
• :livebooks — .livemd notebooks with parsed body and source metadata
• :changelog — release-note entries from the configured source
• :openapi — a valid OpenAPI document
• :presentation — navigation, search, and content indexes

Loading release notes from another store

CHANGELOG.md is only the default source. A host whose documentation lives in a graph, database, CMS, or service implements one callback and returns the same renderer-neutral changelog entries:

defmodule MyApp.Docs.ReleaseNotes do
  @behaviour DocShell.Generate.Changelog.Source

  @impl true
  def load(opts) do
    opts
    |> Keyword.fetch!(:markdown)
    |> DocShell.Generate.Changelog.from_markdown("graph://docs/releases")
  end
end

DocShell.Build.run(
  changelog_source: MyApp.Docs.ReleaseNotes,
  changelog_options: [markdown: "## v1.0.0 (2026-08-28)\n\nInitial release"],
  write: false
)

Sources that already store structured entries can return those maps directly; DocShell validates them before projection or artifact writing.

Inspect module extraction

Module documentation is read from compiled BEAM documentation. That is the same source IEx uses for h DocShell.Config, which keeps the artifact aligned with what developers see locally.

module_entry = Enum.find(result.modules, &(&1["id"] == "DocShell.Config"))

Map.take(module_entry, ["id", "title", "kind"])

The module metadata carries structural information useful for renderers and coverage tooling.

module_entry["meta"]
|> Map.take(["module", "language", "moduledoc"])

Function, callback, and type docs are exposed as member metadata. Member docs stay as Markdown strings; renderers can parse them lazily instead of paying the cost for every member up front.

module_entry["meta"]["members"]
|> Enum.map(&Map.take(&1, ["kind", "name", "arity", "signatures"]))
|> Enum.take(5)

Inspect guide extraction

Guides are Markdown files with optional YAML frontmatter. DocShell passes frontmatter through as metadata and exposes audience and locale as search filters.

guide_entry = List.first(result.guides)

Map.take(guide_entry, ["id", "title", "kind", "meta"])

The parsed body lives under "ast" in the in-memory result.

guide_entry["ast"] |> List.first()

Inspect Livebook extraction

Livebooks are indexed from .livemd files. Unlike guides, they do not use YAML frontmatter; Livebook owns the top of the file, and DocShell takes the id from the filename and the title from the first Markdown H1.

livebook_entry = List.first(result.livebooks)

Map.take(livebook_entry, ["id", "title", "kind", "meta"])

Code cells remain source text in the AST. DocShell does not execute them.

livebook_entry["ast"]
|> Enum.find(fn node ->
  node["tag"] == "pre"
end)

Inspect the OpenAPI output

If no OpenAPI adapter is configured, DocShell still emits a valid empty OpenAPI 3.1 document. That means renderers can always expect openapi.json to exist.

Map.take(result.openapi, ["openapi", "info", "paths"])

Use the OpenAPI adapters notebook for Ash, OpenApiSpex, raw JSON, and custom adapter examples.

Inspect the presentation indexes

The presentation layer turns all extracted entries into three renderer-facing indexes:

• navigation — what to list
• search — what to index
• content — what to render for a page

Map.keys(result.presentation) |> Enum.sort()

Navigation items are structs in memory. The default generator keeps the tree flat because hierarchy is a host product decision.

result.presentation.navigation
|> Enum.map(&Map.take(&1, [:id, :title, :path, :kind, :children]))

Search entries contain the same identity and path information plus flattened plain text content. Tokens are present but empty unless search_tokens: true is set.

result.presentation.search
|> Enum.map(&Map.take(&1, [:id, :title, :path, :kind, :audience, :locale, :tokens]))

Content maps ids to AST node lists. The body is stored once, keyed by id, rather than duplicated in the navigation and search indexes.

Map.keys(result.presentation.content) |> Enum.sort()

Presentation validation also checks relationships: nonempty IDs are unique in navigation (including descendants) and search, shared IDs have the same path, and local leaves/search records have content. Groups with children and absolute HTTP(S) links may omit content. Backlink targets are local; origins may belong to the host's wider graph. Default navigation/search order is kind, title, then ID.

Inspect the written files

Because write defaults to true, the build also wrote an artifact tree.

Path.wildcard(Path.join(public_dir, "*.json"))
|> Enum.map(&Path.basename/1)
|> Enum.sort()

Use DocShell.Artifact.read/1 to read an artifact payload. It validates the envelope before returning the "data" field.

{:ok, manifest} = DocShell.Artifact.read(Path.join(public_dir, "manifest.json"))

Map.take(manifest, ["artifacts"])

The private directory currently receives only its own manifest. Hosts can use the public and private directories with different cache processes and authorization rules.

DocShell.Artifact.read(Path.join(private_dir, "manifest.json"))

Skip file writes when you only need data

Hosts that ingest documentation into a database or knowledge graph can keep the in-memory result and skip the files.

dry_workspace =
  Path.join(
    System.tmp_dir!(),
    "doc_shell_build_pipeline_dry_#{System.unique_integer([:positive])}"
  )

dry_public_dir = Path.join(dry_workspace, "public")
dry_private_dir = Path.join(dry_workspace, "private")

dry_opts =
  build_opts
  |> Keyword.put(:public_dir, dry_public_dir)
  |> Keyword.put(:private_dir, dry_private_dir)
  |> Keyword.put(:write, false)

{:ok, dry_result} = DocShell.Build.run(dry_opts)

%{
  returned_keys: Map.keys(dry_result) |> Enum.sort(),
  wrote_public_dir?: File.exists?(dry_public_dir)
}

Errors from invalid sources

Extraction stops at the first configured source that cannot be read. The error names the module or file at fault; invalid sources are never silently skipped.

DocShell.Build.run(Keyword.put(build_opts, :modules, [Nonexistent.Module]))

Guide frontmatter is treated the same way. A malformed configured guide returns an error instead of disappearing from the documentation set.

bad_guide_dir = Path.join(workspace, "bad-guides")
File.mkdir_p!(bad_guide_dir)

File.write!(Path.join(bad_guide_dir, "broken.md"), """
---
title: Broken

# This frontmatter never closes
""")

DocShell.Build.run(
  build_opts
  |> Keyword.put(:modules, [])
  |> Keyword.put(:guide_bases, [bad_guide_dir])
)

Build configuration checklist

When wiring DocShell into a host application, make these decisions explicitly:

Changelog source validation

Every changelog source entry must be a valid entry map; nil and other invalid entries return {:error, {:invalid_changelog_entry, entry}}.

Guide line endings

YAML frontmatter accepts LF, CRLF, and CR line endings, including a closing --- delimiter at end of file.

Document identities

Document IDs must be nonempty and unique across all sources, including entries filtered from presentation. Duplicate IDs return an error naming both sources. Overlapping guide directories extract each normalized path once.

JSON metadata normalization

Metadata preserves JSON scalars and uses UTF-8 string keys. Unsupported terms become inspected text; improper list tails become a final array value. DocShell.Json.normalize/1 rejects converted-key collisions. The legacy stringify/1 keeps string keys when a collision occurs. Guides use normalize/1.

Custom ExDoc member metadata is checked during normalization: atom/string keys that become the same JSON key fail with the member name, rather than losing data.

Configuration errors

Build.run/1 rejects malformed options and unknown per-call keys before extraction. Unknown application environment keys remain ignored. Invalid guide identities, titles, audience, and locale return errors naming the field and file. Guide IDs and titles accept nonempty strings or numeric/boolean scalars.

Output destinations

Public and private output directories must be disjoint. The optional raw OpenAPI destination must lie outside both. Conflicting paths fail before extraction or writes; use dedicated directories without symlink aliases.

Markdown titles

Guide and notebook titles come from the first top-level parsed H1, including Setext headings. Inline formatting is flattened, and headings inside code examples are ignored. Explicit guide frontmatter titles still take precedence.

Guide title extraction avoids a second parse for explicit frontmatter titles or unfenced bodies. Fenced bodies retain a defensive title-only parse, so malformed closing fences cannot turn code comments into page headings.

Changelog Markdown context

Changelogs are parsed as complete Markdown documents before splitting on top-level release headings. Code examples remain within their release, and reference links resolve across the whole document. Parse errors in any part of the source return a source-tagged error.

Collection builds

The collection integrity requirements in docs/specs/DSH.01-documentation-sites.md cover multi-release changelogs, reserved identities, malformed input, and exact build/load round trips. The implementation plan records acceptance status. Invalid source encoding must produce an error identifying the source file.

Failed builds and recovery

Builds stage all JSON and back up existing files before publishing. Returned publication failures restore earlier files, including a stale collection.json removed by a non-collection build. Artifact.Transaction.write/2 accepts a delete: [path] list covered by the same locks and backups. Rollback failures report retained backup paths.

Cooperating builds use .doc-shell-build.lock directories. After a process or machine crash, recover retained backups and remove stale locks before rebuilding. Files still publish individually, so cache reloads validate generation IDs and keep the last complete snapshot. Use dedicated output directories without external writers or symlink aliases.