Byte-exact top-level form span scanner for raw PTC-Lisp prelude source TEXT.

The public and fast parser implementations track no source positions, and parse -> swap-form -> re-render via Formatter would reformat every untouched form, destroying byte-identity. scan/1 instead walks the RAW TEXT directly, tracking bracket/string/comment depth well enough to compute the byte span of every top-level form plus the whitespace/comment "gap" that precedes it, without ever building an AST.

Result shape

scan/1 returns {:ok, result} where result is:

%{
  forms: [
    %{
      head: "defn",           # see "head/name" below
      name: "get-user",       # nil if the form has no name (see below)
      gap: {0, 12},           # {offset, length} gap BEFORE this form
      span: {12, 84}          # {offset, length} of the form itself
    },
    ...
  ],
  trailing_gap: {96, 4}       # {offset, length} of content after the
                              # LAST form (whitespace/comments/EOF slack)
}

Every offset/length pair is in BYTES, not characters. Reconcatenating gap ++ span ++ gap ++ span ++ ... ++ trailing_gap in order reproduces the original source byte-for-byte — scan/1 checks this itself (see "Self-checks" below), so callers never have to.

head / name

A prelude is not the only source this scanner has to survive: the corpus gate (below) includes test/smoke/*.clj files, which are plain PTC-Lisp scripts, not preludes — their final top-level "form" is often a bare return value (a map literal, a vector, ...), not a (head ...) list. So head/name are defined for ANY top-level value, not only lists:

• If the form is a list whose first element is a symbol ((defn ...), (let ...), (anything-callable ...)), head is that symbol's literal source text.
• name is populated only when head is one of the four prelude-recognized heads — ns, defn, defn-, def — and the list's SECOND element is itself a plain symbol (the namespace name for ns, the def/defn name otherwise). This mirrors PtcRunner.Lisp.Prelude.Compiler's handle_ns_directive/2, handle_defn/3, and handle_def/2, which all take the name from that same position — but this module does NOT call into Compiler; the two are independent by construction (see "Self-checks").
• Every other shape — a list with a non-symbol head (or an empty list), or any non-list top-level value (a vector, map, set, string, number, keyword, ...) — gets a generic head describing its syntactic kind ("list", "vector", "map", "set", "short_fn", "string", "regex_literal", "var", "quoted_symbol", "keyword", "number", "nil", "true", "false") and name: nil. A bare top-level SYMBOL (unusual, but legal) gets its own literal text as head.

head/name are metadata for humans and for prelude/edit to key off of — never re-parsed by this module, and never assumed to be unique.

Design decisions (fail-closed, on purpose)

1. Fails closed on surplus top-level delimiters. The real reader records a stray )/]/} between top-level forms while scanning onward so a later hard syntax failure can remain the primary diagnostic. If no later failure occurs, the reader rejects the recorded closers. This scanner has no diagnostic-recovery role, so it rejects the first stray closer directly as %{reason: :unexpected_byte}.
2. Character literals classify as "string", not "char_literal". The reader itself has no separate character-literal AST node — \a and "a" both parse to {:string, "a"} (see FastParser.parse_char/1 and the language spec §3.5: "Character literals are represented as single-character strings internally"). Giving them a distinct head label here would make this scanner's classification strictly finer than the reader's own value model, which the cross-check (below) would then have no way to agree with. So both scan to "string".
3. No semantic validation. Map literals aren't checked for even arity, number/keyword literals aren't validated beyond their character class, and namespaced/qualified symbols aren't specially parsed — this module only needs byte-accurate token BOUNDARIES, never the parsed VALUE. Full semantic validation still happens at the existing compile gate (PtcRunner.Lisp.Prelude.Compiler.compile/1).

Self-checks (both run INSIDE scan/1, never left to the caller)

1. Round-trip. Reconcatenating every gap + form span + the trailing gap must reproduce the input byte-for-byte, or scanning fails with %{reason: :scan_roundtrip_mismatch}.
2. Cross-check against the real parser. scan/1 also derives the ordered {head, name} list by calling PtcRunner.Lisp.Parser.parse/1 on the SAME source and walking the returned AST (a second, independent code path — see derive_head_name/1 below, which classifies parsed AST shapes the same way tag_label/1 classifies raw scan tags, without either function calling the other). Any disagreement (including the real parser rejecting source this scanner accepted) fails with %{reason: :scan_cross_check_mismatch} or %{reason: :scan_cross_check_parse_failed}. This is also the backstop for lexical-grammar drift: if FastParser's reader grammar ever changes and this hand-written mirror doesn't get updated to match, the cross-check starts failing loudly instead of silently drifting.

With both checks in place, the worst failure mode is a loud refusal — never a silently wrong span.

Locating the (ns ...) docstring

locate_ns_doc/2 is a narrowly-scoped helper for source-authoring tools: given the byte span of a (ns ...) form (as produced by scan/1 — a %{head: "ns", ...} entry), it locates the docstring token, if any, WITHOUT re-deriving or duplicating any of PtcRunner.Lisp.Prelude.Compiler's ns_metadata/1 semantics — it reuses this module's own scan_symbol/1/scan_value/1/skip_ws/1 machinery against the same span. Span in, span(s) out: callers splice bytes themselves; this module never rewrites source text.

Reader-macro coverage

Mirrors every construct the fast parser accepts: lists (...), vectors [...], maps {...}, sets #{...}, short-fn #(...), strings "..." and regex literals #"..." (identical escape rules — \\, \", \n, \t, \r, any other \<char> passed through, and literal embedded newlines allowed; unterminated is the one string-side error), character literals (\newline, \space, \tab, \return, \backspace, \formfeed, or any other single UTF-8 codepoint), var refs #'name, quoted symbols 'name (quoted COLLECTIONS stay unsupported, same as the reader), keywords :name, numbers (int/float/exponent), the ##Inf/##-Inf/##NaN literals, nil/true/false, comma-as-whitespace, ; line comments, and symbols (including the embedded-' prime-notation rest-char, e.g. inc').