# Ichor v0.2.1 - Table of Contents

> Compiles grammar definitions -- Aether, its own DSL, plus ABNF/EBNF/PEG/BNF importers -- into separated Lexer/Parser/Analysis/Executor modules, with both an interpreted VM backend and a compile-time native codegen backend.

## Pages

- [Ichor](readme.md)
- [Ichor Tutorial](tutorial-1.md)
- [Examples](examples.md)
- [Cheatsheet](cheatsheet.md)
- [Changelog](changelog.md)
- [Contributing to Ichor](contribution.md)
- [LICENSE](license.md)

- Aether
  - [Aether Tutorial](tutorial-2.md)
  - [Aether Reference](aether.md)
  - [Aether Examples](aether_examples.md)
  - [Aether Cheatsheet](aether_cheatsheet.md)

## Modules

- [Aether.Eval](Aether.Eval.md): The semantic half of Aether's front-end: turns an `Aether.Reader.Grammar`
CST into a fully resolved `Aether.Grammar`, with every body already
`Grammar.IR`.
- [Aether.Reader](Aether.Reader.md): The pure-syntax half of Aether's front-end: turns `.aether` source (via
`Aether.Lexer`) into a concrete syntax tree that's a faithful record of
what was written -- every pragma, token, and rule definition, in file
order -- with no desugaring and no cross-definition semantics applied.
- [Aether.Reader.Grammar](Aether.Reader.Grammar.md): The CST `Aether.Reader.read/2` produces: grammar-wide pragmas plus
every token/rule definition in file order, each body an
`Aether.Reader.cst()` tree. `Aether.Eval.build/1` is the only
consumer.

- [Grammar.GLR](Grammar.GLR.md): The interpreted graph-structured-stack backend: runs the same
`Grammar.LRTable` SLR(1) table `Grammar.LR` does, but *accepts*
conflicts instead of requiring their absence -- every action in a
conflict cell is taken, forking the parse across a real
`Grammar.GLR.GSS` (node-sharing, not independent per-branch stacks:
two derivations that reach the same state at the same position
merge, which is what keeps this from blowing up combinatorially on a
grammar with only a handful of local conflicts). The GSS-driving loop
itself lives in `Grammar.GLR.Runtime`, shared unchanged with
`Grammar.Native.GLR` -- this module's own job is just building the
table (recompiled on every call, the same "VM interpreted" convention
`Grammar.VM` and `Grammar.LR` already follow) and wrapping its plain
action/goto maps into the closures `Runtime.run/6` expects.
- [Grammar.IR.Custom](Grammar.IR.Custom.md): `@native("Module", "function", dep1, dep2, ...)`: a rule body that
delegates to hand-written Elixir code (an `Ichor.CustomRule`
implementation) instead of ordinary `Grammar.IR` combinators -- the
escape hatch for constructs no static PEG grammar can express (a
Prolog-style mutable operator-precedence table, C's typedef-vs-
expression ambiguity, and similar "what matches here depends on
something read earlier" problems).
- [Grammar.IR.CustomLexeme](Grammar.IR.CustomLexeme.md): `@native("Module", "function", dep1, ...)` at *token* position: a
token whose entire body is hand-written Elixir code (an
`Ichor.CustomLexeme` implementation) instead of ordinary character-level
combinators -- the escape hatch for lexical constructs no fixed
maximal-munch tokenizer can express (a heredoc's dynamic terminator, a
string literal with embedded interpolated expressions).
- [Grammar.LR](Grammar.LR.md): The deterministic bottom-up backend: builds an SLR(1) table via
`Grammar.LRTable` and requires it be conflict-free -- a real conflict
is a compile-time error here (`compile/1`), never something forked
through (that's what `Grammar.GLR` is for). No graph-structured-stack
bookkeeping at all: an ordinary shift-reduce loop over a single stack.
- [Grammar.LRTable.Automaton](Grammar.LRTable.Automaton.md): Canonical LR(0) item-set construction (closure/goto over
`Grammar.LRTable.Desugar`'s flat production list), plus SLR(1)
action/goto table construction on top of it.
- [Grammar.LRTable.Builder](Grammar.LRTable.Builder.md): Builds an SLR(1) action/goto table (`Grammar.LRTable`, from
`ichor_runtime`) from an `@engine lr`/`@engine glr` grammar -- shared
by both `Grammar.LR` (which additionally requires the result be
conflict-free) and `Grammar.GLR` (which accepts conflicts and forks
over them at runtime).
- [Grammar.LRTable.Desugar](Grammar.LRTable.Desugar.md): Flattens `grammar.rules`' PEG-shaped IR into the flat CFG production
list `Grammar.LRTable.Production` describes, for `Grammar.LRTable`'s
LR(0)/SLR(1) construction to consume.
- [Grammar.LRTable.Sets](Grammar.LRTable.Sets.md): Standard textbook nullable/FIRST/FOLLOW fixpoint algorithms, over
`Grammar.LRTable.Desugar`'s flat CFG production list rather than the
original PEG IR -- recomputed fresh (not reusing `Grammar.Analysis`'s
own `compute_nullable/1`) since the augmented nonterminal set here
includes every helper nonterminal a `Star`/`Plus`/`Opt`/`Rep`/group
desugared into, which `Grammar.Analysis` never sees.

- [Grammar.Native.GLR](Grammar.Native.GLR.md): The compile-time codegen backend for `@engine glr` grammars: builds
the SLR(1) table via `Grammar.LRTable` once, at Elixir-compile-time
(conflicts are *expected* here, never rejected -- that's
`Grammar.Native.LR`'s job), and compiles the action/goto lookup into
per-state generated function clauses instead of the interpreted
`Grammar.GLR`'s `Map.get`s.
- [Grammar.Native.LR](Grammar.Native.LR.md): The compile-time codegen backend for `@engine lr` grammars: builds the
SLR(1) table via `Grammar.LRTable` once, at Elixir-compile-time (inside
`use Ichor`'s macro expansion), and requires it be conflict-free --
same requirement `Grammar.LR.compile/1` enforces at runtime, just
checked once here instead of on every call.
- [Grammar.Native.TokenizerCompiler](Grammar.Native.TokenizerCompiler.md): Generates the Tokenizer -> Lexer half of a compiled grammar's `tokenize/2`
(char-level maximal munch via `Grammar.Native.CharCompiler`, then
`@keywords`/`@refine` reclassification via the shared, backend-agnostic
`Grammar.Lexer`) -- extracted once out of `Grammar.Native.generate/2` so
`Grammar.Native.LR`/`Grammar.Native.GLR` can reuse it unchanged: every
engine consumes the exact same token stream, only the *parser* half
differs (direct PEG combinator calls, vs. compiled LR/GLR state
dispatch).
- [Grammar.VM.Tokenizer](Grammar.VM.Tokenizer.md): Tokenizes a full input string against a `Grammar.VM.CharCompiler`
program, using maximal munch: at each position, every declared token is
tried, the longest match wins, ties are broken by declaration order.
Trivia/skip tokens are emitted like any other -- filtering them out is
the *parser*'s job, via the `Star(skip_token)` that `Aether.Parser`
already spliced into rule bodies, not the lexer's.
- [Ichor.TokenRefiner](Ichor.TokenRefiner.md): The behaviour a `@refine("Module", "function", ...)` token suffix
dispatches to -- reclassifying (or validating/decoding) a token the
Tokenizer already matched, before the Parser ever sees it. `@keywords`
is sugar for the common table-lookup case of this same mechanism (see
`Grammar.Lexer`); `@refine` is the escape hatch for anything needing
real logic: escape-sequence decoding, or disambiguating a token based
on what came immediately before it (JS's `/` being a regex-literal
start or a division operator, depending on the preceding token).
- [Ichor.Toolkit.Codegen](Ichor.Toolkit.Codegen.md): Small, IR-agnostic `quote`/`unquote` mechanics -- not a code
generator, and deliberately not a framework for building one (no
tree-walker, no visitor dispatch, no assumed node shape, no assumed
function-signature convention). Each function here solves exactly one
mechanical pain point that recurs when hand-writing a compiler-to-
quoted-Elixir backend, extracted from `Grammar.Native`'s own codegen
(`CharCompiler`, `RuleCompiler`, `Native.LR`) after the same handful
of tricks turned up independently duplicated across (and within) those
modules -- the same duplication signal that justified
`Ichor.Toolkit.Fixpoint`.

- [Ichor.Toolkit.Fixpoint](Ichor.Toolkit.Fixpoint.md): The generic "repeatedly recompute a value from itself until it stops
changing" primitive -- extracted from a pattern that had already been
hand-written three separate times in Ichor's own compiler internals
before this existed: `Grammar.Analysis`'s private `fixpoint/3`
(nullable-set and always-empty-set computation, both over a
`MapSet.t(atom())` of rule/token names) and `Grammar.LRTable.Sets`'
private `fixpoint/2`/`fixpoint_map/2` (FIRST/FOLLOW-set computation,
one over a `MapSet`, one over a `Map`). None of those three needed
anything MapSet- or Map-specific about the *iteration* itself -- only
about what `step_fn` does with the value in between -- which is
exactly what this generalizes: `step_fn` closes over whatever fixed
data it needs (a grammar's own rule list, a graph, ...), this module
never sees that data at all, only the value being iterated.
- [Ichor.Toolkit.Graph](Ichor.Toolkit.Graph.md): Small, generic graph primitives -- extracted from `Grammar.Analysis`'s
own private `reachable/2`/`do_reachable/3` (used to detect left
recursion: a rule is left-recursive exactly when it's reachable from
its own leading references). Nothing here is grammar-specific --
`neighbors_fn` is the caller's own adjacency, a plain function rather
than a hardcoded map, so this works over whatever graph an author's
own AST/IR/dependency structure represents, not just Ichor's own
rule-reference graphs.

- [Ichor.Toolkit.Layout](Ichor.Toolkit.Layout.md): The off-side-rule (indentation-sensitive layout) algorithm Python's
own tokenizer uses, and Haskell's/F#'s: given each logical line's
indentation width, in order, maintain a stack of open indentation
levels and emit `:indent`/`:dedent` markers as the width rises or
falls -- so a parser downstream never has to know about columns at
all, just consumes ordinary tokens.
- [Ichor.Toolkit.Scope](Ichor.Toolkit.Scope.md): A generic nested lexical scope / symbol table: a stack of name->value
bindings, innermost scope first, supporting the "define here, look up
through enclosing scopes" shape almost every real language's semantic
analysis needs (variable resolution, nested function/block scoping,
...). No existing internal precedent to extract this from (unlike
`Ichor.Toolkit.Fixpoint`/`Graph`) -- a from-scratch design, but a
standard, well-understood one.
- [Ichor.Toolkit.TypeScheme](Ichor.Toolkit.TypeScheme.md): Hindley-Milner-style let-polymorphism: `generalize/4` and
`instantiate/3`, built on `Ichor.Backtrack.Bindings.unify_occurs_check/4`
(plain `unify/4` deliberately has no occurs-check, matching ISO
Prolog's own default -- a type checker needs one, or unifying a type
variable with a type containing itself would silently build an
infinite type instead of failing).

- Core
  - [Ichor](Ichor.md): Ichor reads grammar definitions -- its own language Aether, plus ABNF,
BNF, EBNF, and PEG importers -- and turns each one into a working
Lexer + Parser + Executor for whatever language the grammar describes,
via either an interpreted VM backend (`Grammar.VM`) or compile-time
native codegen (`__using__/1` below).
  - [Ichor.GrammarImport](Ichor.GrammarImport.md): Turns an ABNF/BNF/ISO-EBNF/PEG ruleset (as `Ichor.ABNF`/`Ichor.BNF`/
`Ichor.EBNF.ISO`/`Ichor.PEG` produce them -- `%{rule_name_atom =>
Grammar.IR.expr()}`, explicitly documented as "not a runnable
`Aether.Grammar`") into one that actually is: picks a root (the
source's own first-declared rule, unless overridden), assembles a
full `%Aether.Grammar{}`, and (for ABNF specifically) fills in RFC
5234 Appendix B's implicit "core rules" (`ALPHA`, `DIGIT`, `CRLF`,
...) for anything the source references but never defines itself --
extremely common in real ABNF, which generally assumes those are
always available.

- Grammar IR
  - [Grammar.IR](Grammar.IR.md): The normalized grammar AST every Ichor front-end (Aether, ABNF, BNF,
EBNF, PEG) compiles down to, and that every later stage -- the analysis
pass, `Grammar.VM`, native codegen -- consumes instead of talking to any
particular front-end's own syntax.
  - [Grammar.IR.AndPred](Grammar.IR.AndPred.md): Positive lookahead, consumes nothing: `&expr`.
  - [Grammar.IR.Any](Grammar.IR.Any.md): Matches any single character: `.`.
  - [Grammar.IR.Capture](Grammar.IR.Capture.md): Named capture for AST construction: `name:expr`.
  - [Grammar.IR.CharClass](Grammar.IR.CharClass.md): Character class, including unicode ranges: `[a-z0-9_]`.
  - [Grammar.IR.Choice](Grammar.IR.Choice.md): Ordered PEG choice: `a | b | c` -- first match wins, always.
  - [Grammar.IR.Indent](Grammar.IR.Indent.md): Indentation-sensitive block: `@indent(expr)` / `@samecol(expr)`.
  - [Grammar.IR.Literal](Grammar.IR.Literal.md): Exact string match.
  - [Grammar.IR.Meta](Grammar.IR.Meta.md): Source-location metadata carried by every `Grammar.IR` node, so an error
raised at any later stage (analysis, either backend) can still point
back at the exact line/column in the original grammar file rather than
just naming an IR node in the abstract.

  - [Grammar.IR.NotPred](Grammar.IR.NotPred.md): Negative lookahead, consumes nothing: `!expr`.
  - [Grammar.IR.Opt](Grammar.IR.Opt.md): Zero or one: `expr?`.
  - [Grammar.IR.Plus](Grammar.IR.Plus.md): One or more: `expr+`.
  - [Grammar.IR.Rep](Grammar.IR.Rep.md): Bounded repetition: `expr{n}`, `expr{n,}`, `expr{n,m}`.
  - [Grammar.IR.RuleRef](Grammar.IR.RuleRef.md): Reference to another rule (or token), by name.
  - [Grammar.IR.Seq](Grammar.IR.Seq.md): Ordered sequence of sub-expressions: `a b c`.
  - [Grammar.IR.Star](Grammar.IR.Star.md): Zero or more: `expr*`.

- Aether Front-end
  - [Aether.Grammar](Aether.Grammar.md): The fully compiled output of `Aether.Parser`: grammar-wide settings plus
every token and rule body, each already resolved to `Grammar.IR`.
  - [Aether.Lexer](Aether.Lexer.md): Turns `.aether` source text into a flat list of `Aether.Token`s.
  - [Aether.Parser](Aether.Parser.md): Parses `.aether` source into a fully resolved `Aether.Grammar`, in two
stages: `Aether.Reader` (pure syntax -- a token/rule reference split,
pragmas, the expression grammar -- into a concrete syntax tree with no
desugaring) followed by `Aether.Eval` (that CST into `Grammar.IR`,
handling everything that needs whole-grammar context: predefined-token
override/use tracking, case-insensitivity, character-class/regex
desugaring, inline-literal promotion, `@skip` splicing, and final
`@root`/`@skip` validation).
  - [Aether.Token](Aether.Token.md): A single lexical token produced by `Aether.Lexer` from `.aether` source
text.

- Analysis
  - [Grammar.Analysis](Grammar.Analysis.md): The analysis pass between a front-end (`Aether.Parser`, or one of the
ABNF/BNF/EBNF/PEG importers) and either backend (`Grammar.VM` or native
codegen)

- VM Backend
  - [Grammar.VM](Grammar.VM.md): The interpreted runtime backend: compiles an `Aether.Grammar` to
bytecode and runs it against real input. No compile-time codegen step
-- give it a grammar (from `Aether.Parser.parse/2` +
`Grammar.Analysis.run/1`) whenever you have one, including one your
own program only learns about at runtime (a user-supplied grammar, a
plugin, a REPL's `:load`). See the tutorial's "Loading a grammar at
runtime" section for a full worked example, and "Which path is right
for you?" for how this compares to `use Ichor`/`Mix.Tasks.Ichor.Gen`.
  - [Grammar.VM.CharCompiler](Grammar.VM.CharCompiler.md): Compiles every token in a grammar (character-level: `Literal`,
`CharClass`, `Any`, and `RuleRef` to another token -- the only node
shapes an *ordinary* token body can contain) into one linked
`Grammar.VM.Program`, run by `Grammar.VM.Tokenizer`.
  - [Grammar.VM.CharInterpreter](Grammar.VM.CharInterpreter.md): Runs a `Grammar.VM.CharCompiler`-produced program against raw input
text -- the machine `Grammar.VM.Tokenizer` uses to test one token at a
given starting position.
  - [Grammar.VM.Compiler](Grammar.VM.Compiler.md): The combinator half of Grammar.VM's bytecode compilation, shared
between the character-level compiler (token bodies, run by the lexer)
and the token-stream-level compiler (rule bodies, run by the parser).
`Seq`/`Choice`/`Star`/`Plus`/`Opt`/`Rep`/`AndPred`/`NotPred` compile
identically either way -- only the leaf nodes differ
(`Literal`/`CharClass`/`Any`/`RuleRef` for tokens; `RuleRef`/`Indent`/
`Capture` for rules -- tokens can never contain a `Capture`), so each
caller supplies its own `leaf_fun` for those.
  - [Grammar.VM.Linker](Grammar.VM.Linker.md): Turns `[{name, ops}]` -- one label-relative op list per token or rule,
as produced by `Grammar.VM.Compiler` -- into a single linked
`Grammar.VM.Program`.
  - [Grammar.VM.Program](Grammar.VM.Program.md): Linked PEG bytecode: every token (or every rule) in a grammar, compiled
into one shared instruction tuple so `:call` can jump between them by
index, plus a name -> entry-index map for looking up where to start.
  - [Grammar.VM.RuleCompiler](Grammar.VM.RuleCompiler.md): Compiles every rule in a grammar into one linked `Grammar.VM.Program`,
run by `Grammar.VM.TokenInterpreter` against the lexer's token stream
(never against raw characters -- Aether's two-stage Lexer -> Parser
split). A `RuleRef` compiles to `{:token, name}` (consume one stream
token of that type) when it names a token, or `{:call, name}` (jump
into that rule's own compiled code) when it names another rule -- the
two are otherwise indistinguishable in the IR, so the grammar's own
token/rule namespaces are what disambiguate.
  - [Grammar.VM.TokenInterpreter](Grammar.VM.TokenInterpreter.md): Runs a `Grammar.VM.RuleCompiler`-produced program against a lexed token
stream -- the machine `Grammar.VM` uses for the parser stage.

- Native Backend
  - [Grammar.Native](Grammar.Native.md): The compile-time codegen backend: turns a validated `%Aether.Grammar{}`
into quoted Elixir function definitions -- the same two-stage Lexer ->
Parser split `Grammar.VM` compiles to bytecode for, but here as direct
function calls a `use Ichor, grammar:, actions:` caller splices
straight into its own module (via `Ichor`'s `__using__` macro),
skipping bytecode interpretation entirely.
  - [Grammar.Native.CharCompiler](Grammar.Native.CharCompiler.md): Compiles every token in a grammar (character-level: `Literal`,
`CharClass`, `Any`, `RuleRef` to another token, and the combinators
`Seq`/`Choice`/`Star`/`Plus`/`Opt`/`Rep`/`AndPred`/`NotPred` -- the
same shapes `Grammar.VM.CharCompiler` compiles) into quoted Elixir
function definitions instead of bytecode.
  - [Grammar.Native.RuleCompiler](Grammar.Native.RuleCompiler.md): Compiles every rule in a grammar into quoted Elixir function
definitions run against the lexer's token stream (never raw
characters -- Aether's own two-stage split), mirroring
`Grammar.VM.RuleCompiler`'s own semantics exactly (bare-reference
implicit self-capture, `Indent`/`@samecol`, capture-shape rules) but
producing direct function calls instead of bytecode.

- Tooling
  - [Grammar.Tokens](Grammar.Tokens.md): Token introspection for a compiled `Aether.Grammar` -- the logic behind
`mix ichor.tokens`. Depends only on the Aether front-end's output
(`token_order`/`tokens`/`anon_tokens`), not the analysis pass or either
backend, so it works on any grammar that parses, even one the analysis
pass would otherwise reject.

- Format Importers
  - [Ichor.ABNF](Ichor.ABNF.md): The ABNF (RFC 5234 + RFC 7405) front-end: `priv/grammar/abnf.aether`
parsed by Aether's own front-end, compiled by the native codegen
backend (`use Ichor`), dispatching to `Ichor.ABNF.Actions`
(`Grammar.IR` as its target category, same as `Regex.Actions`).
  - [Ichor.ABNF.Actions](Ichor.ABNF.Actions.md): Turns a parsed ABNF `rulelist` (RFC 5234 + RFC 7405) into a real
`%{rule_name_atom => Grammar.IR.expr()}` map -- one `Grammar.IR` tree
per ABNF rule, mirroring how `Regex.Actions` turns a single
`/pattern/` into one `Grammar.IR` tree. This is an importer's own
output, not a runnable `Aether.Grammar` -- ABNF has no lexer/rule
(token/parser) split the way Aether does, so deciding which of an
ABNF ruleset's productions become Aether tokens vs. rules is a
separate concern this module doesn't address.
  - [Ichor.BNF](Ichor.BNF.md): The classical BNF front-end -- no single citable standard exists;
this follows the ALGOL 60 Report's own convention with quoted
terminals, matching how BNF is actually written today (see
`Ichor.BNF.Actions`'s own moduledoc for the deliberate deviation that
is): `priv/grammar/bnf.aether` parsed by Aether's own front-end,
compiled by the native codegen backend (`use Ichor`), dispatching to
`Ichor.BNF.Actions` (`Grammar.IR` as its target category).
  - [Ichor.BNF.Actions](Ichor.BNF.Actions.md): Turns a parsed classical BNF `grammar_file` into a real
`%{nonterminal_name_atom => Grammar.IR.expr()}` map -- one
`Grammar.IR` tree per rule, the same target category `Ichor.ABNF.Actions`
and `Regex.Actions` both use.
  - [Ichor.EBNF.ISO](Ichor.EBNF.ISO.md): The ISO/IEC 14977 EBNF front-end: `priv/grammar/ebnf-iso.aether`
parsed by Aether's own front-end, compiled by the native codegen
backend (`use Ichor`), dispatching to `Ichor.EBNF.ISO.Actions`
(`Grammar.IR` as its target category).
  - [Ichor.EBNF.ISO.Actions](Ichor.EBNF.ISO.Actions.md): Turns a parsed ISO/IEC 14977 EBNF `syntax` into a real
`%{meta_identifier_atom => Grammar.IR.expr()}` map -- one `Grammar.IR`
tree per rule, the same target category `Ichor.ABNF.Actions` and
`Ichor.BNF.Actions` both use.
  - [Ichor.EBNF.W3C](Ichor.EBNF.W3C.md): The W3C-style EBNF front-end -- the notation the XML 1.0 spec's own
section 6 "Notation" uses, also shared by XQuery/XPath's grammars:
`priv/grammar/ebnf-w3c.aether` parsed by Aether's own front-end,
compiled by the native codegen backend (`use Ichor`), dispatching to
`Ichor.EBNF.W3C.Actions` (`Grammar.IR` as its target category).
  - [Ichor.EBNF.W3C.Actions](Ichor.EBNF.W3C.Actions.md): Turns a parsed W3C-style EBNF `grammar_file` (the notation the XML 1.0
spec's own section 6 uses, also shared by XQuery/XPath's grammars)
into a real `%{ident_atom => Grammar.IR.expr()}` map -- one
`Grammar.IR` tree per rule, the same target category every other
importer this project builds uses.
  - [Ichor.PEG](Ichor.PEG.md): The PEG front-end (Ford's paper, pest/PEG.js-style convention):
`priv/grammar/peg.aether` parsed by Aether's own front-end, compiled
by the native codegen backend (`use Ichor`), dispatching to
`Ichor.PEG.Actions` (`Grammar.IR` as its target category).
  - [Ichor.PEG.Actions](Ichor.PEG.Actions.md): Turns a parsed PEG `grammar_file` (Ford's paper, pest/PEG.js-style
convention) into a real `%{ident_atom => Grammar.IR.expr()}` map --
one `Grammar.IR` tree per rule, the same target category every other
importer this project builds uses.

## Mix Tasks

- Tooling
  - [mix ichor.gen](Mix.Tasks.Ichor.Gen.md): Runs the exact same parse -> analyze -> native-codegen pipeline as
`use Ichor, grammar: ..., actions: ...` (`Ichor.generate/3`), but once,
from the command line, writing the result to disk as an ordinary
module instead of splicing it into a macro expansion. **This is the
recommended way to use Ichor for anything shipping to production** --
see below for why.
  - [mix ichor.tokens](Mix.Tasks.Ichor.Tokens.md): Reads a `.aether` grammar file and lists every token it defines --
user-declared, anonymous (auto-promoted from an inline rule literal),
and the five predefined tokens (always present whether overridden or
left at their default) -- in `token_order`, the same order the Lexer's
maximal-munch tie-break actually uses, alongside a rendered pattern for
each (`Grammar.Tokens`).

