# Ichor v0.1.1 - Table of Contents

> Compiles grammar definitions -- Aether, its own DSL, plus ABNF/EBNF/PEG/BNF importers -- into separated Lexer/Parser/Analysis/Executor modules, with both an interpreted VM backend and a compile-time native codegen backend.

## Pages

- [Ichor](readme.md)
- [Ichor Tutorial](tutorial-1.md)
- [Examples](examples.md)
- [Cheatsheet](cheatsheet.md)
- [Changelog](changelog.md)
- [Contributing to Ichor](contribution.md)
- [LICENSE](license.md)

- Aether
  - [Aether Tutorial](tutorial-2.md)
  - [Aether Reference](aether.md)
  - [Aether Examples](aether_examples.md)
  - [Aether Cheatsheet](aether_cheatsheet.md)

## Modules

- [Aether.Eval](Aether.Eval.md): The semantic half of Aether's front-end: turns an `Aether.Reader.Grammar`
CST into a fully resolved `Aether.Grammar`, with every body already
`Grammar.IR`.
- [Aether.Reader](Aether.Reader.md): The pure-syntax half of Aether's front-end: turns `.aether` source (via
`Aether.Lexer`) into a concrete syntax tree that's a faithful record of
what was written -- every pragma, token, and rule definition, in file
order -- with no desugaring and no cross-definition semantics applied.
- [Aether.Reader.Grammar](Aether.Reader.Grammar.md): The CST `Aether.Reader.read/2` produces: grammar-wide pragmas plus
every token/rule definition in file order, each body an
`Aether.Reader.cst()` tree. `Aether.Eval.build/1` is the only
consumer.

- [Grammar.GLR](Grammar.GLR.md): The interpreted graph-structured-stack backend: runs the same
`Grammar.LRTable` SLR(1) table `Grammar.LR` does, but *accepts*
conflicts instead of requiring their absence -- every action in a
conflict cell is taken, forking the parse across a real
`Grammar.GLR.GSS` (node-sharing, not independent per-branch stacks:
two derivations that reach the same state at the same position
merge, which is what keeps this from blowing up combinatorially on a
grammar with only a handful of local conflicts). The GSS-driving loop
itself lives in `Grammar.GLR.Runtime`, shared unchanged with
`Grammar.Native.GLR` -- this module's own job is just building the
table (recompiled on every call, the same "VM interpreted" convention
`Grammar.VM` and `Grammar.LR` already follow) and wrapping its plain
action/goto maps into the closures `Runtime.run/6` expects.
- [Grammar.GLR.GSS](Grammar.GLR.GSS.md): The graph-structured stack itself: nodes keyed by `{state,
stream_position}` (deduplicated -- two derivations that reach the same
state at the same position share one node, the actual efficiency win
over independent parallel stacks), edges labeled with whatever that
edge's own transition produced (a shifted `%Grammar.VM.Token{}`, or a
reduced nonterminal's own already-built captures map) plus the
provenance accumulated along it so far.
- [Grammar.GLR.Runtime](Grammar.GLR.Runtime.md): The GSS-driving shift-reduce/fork loop itself, shared by the
interpreted `Grammar.GLR` (which wraps a `Grammar.LRTable`'s plain
action/goto maps into closures) and `Grammar.Native.GLR` (which
passes compiled per-state functions instead) -- extracted so both
reuse the exact same, already-proven algorithm. The graph-structured
stack itself (node merging, multi-path reduce enumeration --
`Grammar.GLR.GSS`) is inherently a runtime, input-driven data
structure that can't compile away no matter which engine calls into
it; only *how a caller looks up actions/goto for a state* varies.

- [Grammar.IR.Custom](Grammar.IR.Custom.md): `@native("Module", "function", dep1, dep2, ...)`: a rule body that
delegates to hand-written Elixir code (an `Ichor.CustomRule`
implementation) instead of ordinary `Grammar.IR` combinators -- the
escape hatch for constructs no static PEG grammar can express (a
Prolog-style mutable operator-precedence table, C's typedef-vs-
expression ambiguity, and similar "what matches here depends on
something read earlier" problems).
- [Grammar.IR.CustomLexeme](Grammar.IR.CustomLexeme.md): `@native("Module", "function", dep1, ...)` at *token* position: a
token whose entire body is hand-written Elixir code (an
`Ichor.CustomLexeme` implementation) instead of ordinary character-level
combinators -- the escape hatch for lexical constructs no fixed
maximal-munch tokenizer can express (a heredoc's dynamic terminator, a
string literal with embedded interpolated expressions).
- [Grammar.LR](Grammar.LR.md): The deterministic bottom-up backend: builds an SLR(1) table via
`Grammar.LRTable` and requires it be conflict-free -- a real conflict
is a compile-time error here (`compile/1`), never something forked
through (that's what `Grammar.GLR` is for). No graph-structured-stack
bookkeeping at all: an ordinary shift-reduce loop over a single stack.
- [Grammar.LR.Stack](Grammar.LR.Stack.md): The linear LR stack's shift/reduce mechanics -- shared by the
interpreted `Grammar.LR` and `Grammar.Native.LR`'s per-state
generated code, since a stack push/pop is exactly the same operation
regardless of whether the state driving it came from a `Map.get` or a
compiled `case`.
- [Grammar.LRTable](Grammar.LRTable.md): Builds an SLR(1) action/goto table from an `@engine lr`/`@engine glr`
grammar -- shared by both `Grammar.LR` (which additionally requires the
result be conflict-free) and `Grammar.GLR` (which accepts conflicts and
forks over them at runtime).
- [Grammar.LRTable.Automaton](Grammar.LRTable.Automaton.md): Canonical LR(0) item-set construction (closure/goto over
`Grammar.LRTable.Desugar`'s flat production list), plus SLR(1)
action/goto table construction on top of it.
- [Grammar.LRTable.Captures](Grammar.LRTable.Captures.md): Builds the raw-capture map `Ichor.Actions` expects (`{:token,...}`/
`{:rule,...}`/`{:text,...}`) from one reduced production's own RHS
entries -- shared by `Grammar.LR` (a single linear stack) and
`Grammar.GLR` (a graph-structured one): both reduce the same kind of
production, against the same kind of token stream, needing the exact
same per-position classification (see `Grammar.LRTable.Production`'s
own moduledoc for what each capture kind means), so this logic has
exactly one place to live rather than two copies drifting apart.

- [Grammar.LRTable.Desugar](Grammar.LRTable.Desugar.md): Flattens `grammar.rules`' PEG-shaped IR into the flat CFG production
list `Grammar.LRTable.Production` describes, for `Grammar.LRTable`'s
LR(0)/SLR(1) construction to consume.
- [Grammar.LRTable.Production](Grammar.LRTable.Production.md): One flat CFG production, `lhs -> rhs`, produced by `Grammar.LRTable.Desugar`
from a grammar's PEG-shaped IR (`Seq`/`Choice`/`Star`/`Plus`/`Opt`/`Rep`
each desugar to one or more of these; `Choice` in particular maps
directly onto "more than one production for the same `lhs`" -- the
natural CFG shape ordered PEG choice already resembles, minus the
ordering).
- [Grammar.LRTable.Sets](Grammar.LRTable.Sets.md): Standard textbook nullable/FIRST/FOLLOW fixpoint algorithms, over
`Grammar.LRTable.Desugar`'s flat CFG production list rather than the
original PEG IR -- recomputed fresh (not reusing `Grammar.Analysis`'s
own `compute_nullable/1`) since the augmented nonterminal set here
includes every helper nonterminal a `Star`/`Plus`/`Opt`/`Rep`/group
desugared into, which `Grammar.Analysis` never sees.

- [Grammar.Lexer](Grammar.Lexer.md): The Lexer stage of Aether's Reader/Tokenizer/Lexer/Parser split: walks
a Tokenizer's raw token stream left-to-right, applying every
`@keywords`/`@refine` rule an `Aether.Grammar`'s `refiners` map
declares, before the Parser ever runs.
- [Grammar.Native.GLR](Grammar.Native.GLR.md): The compile-time codegen backend for `@engine glr` grammars: builds
the SLR(1) table via `Grammar.LRTable` once, at Elixir-compile-time
(conflicts are *expected* here, never rejected -- that's
`Grammar.Native.LR`'s job), and compiles the action/goto lookup into
per-state generated function clauses instead of the interpreted
`Grammar.GLR`'s `Map.get`s.
- [Grammar.Native.LR](Grammar.Native.LR.md): The compile-time codegen backend for `@engine lr` grammars: builds the
SLR(1) table via `Grammar.LRTable` once, at Elixir-compile-time (inside
`use Ichor`'s macro expansion), and requires it be conflict-free --
same requirement `Grammar.LR.compile/1` enforces at runtime, just
checked once here instead of on every call.
- [Grammar.Native.Runtime.Parser](Grammar.Native.Runtime.Parser.md): Rule-level helpers shared by every `Grammar.Native.RuleCompiler`-generated
function -- the Parser stage of Aether's Reader/Tokenizer/Lexer/Parser
split. Mirrors `Grammar.VM.TokenInterpreter`'s own semantics exactly
(same backtracking discipline, same no-progress guard on `Star`), just
expressed as plain function composition instead of a bytecode
interpreter loop -- `Grammar.VM.Compiler`'s moduledoc describes the
same combinators this module implements.
- [Grammar.Native.Runtime.Tokenizer](Grammar.Native.Runtime.Tokenizer.md): Char-level helpers shared by every `Grammar.Native.CharCompiler`-generated
function, plus the maximal-munch driver `Grammar.Native.generate/2`'s own
`lex_candidates/2` feeds into -- the Tokenizer stage of Aether's
Reader/Tokenizer/Lexer/Parser split. No captures, no `ref_stack` --
tokens can never contain a `Capture` or `Indent`, so a token body is
just text in, `{matched text, rest}` out.

- [Grammar.Native.TokenizerCompiler](Grammar.Native.TokenizerCompiler.md): Generates the Tokenizer -> Lexer half of a compiled grammar's `tokenize/2`
(char-level maximal munch via `Grammar.Native.CharCompiler`, then
`@keywords`/`@refine` reclassification via the shared, backend-agnostic
`Grammar.Lexer`) -- extracted once out of `Grammar.Native.generate/2` so
`Grammar.Native.LR`/`Grammar.Native.GLR` can reuse it unchanged: every
engine consumes the exact same token stream, only the *parser* half
differs (direct PEG combinator calls, vs. compiled LR/GLR state
dispatch).
- [Grammar.Source](Grammar.Source.md): The very first, Reader-adjacent stage in front of the Tokenizer:
confirms `input` is valid UTF-8 before any char-level matcher --
built on Elixir binary pattern matching against `::utf8` codepoints --
ever touches it. Malformed input left unchecked doesn't fail cleanly:
it crashes a compiled matcher with a `MatchError` partway through
tokenizing, deep inside either backend, instead of surfacing as an
ordinary `Ichor.Error`.
- [Grammar.VM.Tokenizer](Grammar.VM.Tokenizer.md): Tokenizes a full input string against a `Grammar.VM.CharCompiler`
program, using maximal munch: at each position, every declared token is
tried, the longest match wins, ties are broken by declaration order.
Trivia/skip tokens are emitted like any other -- filtering them out is
the *parser*'s job, via the `Star(skip_token)` that `Aether.Parser`
already spliced into rule bodies, not the lexer's.
- [Ichor.Backtrack](Ichor.Backtrack.md): The behaviour a lazy backtracking search engine implements --
`Ichor.Backtrack.Tree` (Stream-backed, correctness-first) is the only
one today; a future WAM-grade engine would implement the same
contract, letting whatever's built on top (a Prolog-style
SLD-resolution loop, say) swap engines without changing.
- [Ichor.Backtrack.Bindings](Ichor.Backtrack.Bindings.md): A substitution: which logic variables are bound to which terms so
far, opaque outside this module. `unify/4` is the whole unification
algorithm -- standard, recursive, structural unification with no
occurs-check (matches ISO Prolog's own default: `X = f(X)` succeeds,
building a cyclic term, rather than failing or looping to detect the
cycle -- occurs-check is opt-in in real Prolog too, via `unify_with_occurs_check/2`,
never the default).
- [Ichor.Backtrack.Term](Ichor.Backtrack.Term.md): The behaviour a caller's own term representation implements so
`Ichor.Backtrack.Bindings` can unify it -- passed explicitly to every
`Bindings` function (`resolve/3`, `unify/4`), the same "module passed
as a plain argument, never looked up dynamically" convention
`Ichor.Actions.evaluate/5` already uses for an actions module.
- [Ichor.Backtrack.Tree](Ichor.Backtrack.Tree.md): The correctness-first `Ichor.Backtrack` engine: a solution is a plain
thunk, `(-> :empty | {:solution, value, rest})` -- a manually-unfolded
lazy list (a "search tree," hence the name), not backed by any
process/agent or Elixir `Stream` machinery. Each combinator forces
exactly one step of whatever it's combining, which is what makes
`once/1` genuinely stop exploring rather than compute everything and
discard all but the first result, and what makes an infinite Prolog
recursion still yield its first solutions instead of hanging forever
building a list.
- [Ichor.CustomLexeme](Ichor.CustomLexeme.md): The behaviour a `Grammar.IR.CustomLexeme` `@native("Module",
"function", ...)` node (at *token* position) dispatches to -- the
escape hatch for lexical constructs a fixed maximal-munch tokenizer
can't express: a heredoc's dynamic terminator, a string literal with
embedded interpolated expressions, a `\catcode`-style mid-scan
reconfiguration.
- [Ichor.CustomRule](Ichor.CustomRule.md): The behaviour a `Grammar.IR.Custom` `@native("Module", "function", ...)`
node dispatches to -- the escape hatch for grammar constructs no static
PEG grammar can express: a Prolog-style mutable operator-precedence
table, C's typedef-vs-expression ambiguity, and similar "what matches
here depends on something read earlier" problems.
- [Ichor.TokenRefiner](Ichor.TokenRefiner.md): The behaviour a `@refine("Module", "function", ...)` token suffix
dispatches to -- reclassifying (or validating/decoding) a token the
Tokenizer already matched, before the Parser ever sees it. `@keywords`
is sugar for the common table-lookup case of this same mechanism (see
`Grammar.Lexer`); `@refine` is the escape hatch for anything needing
real logic: escape-sequence decoding, or disambiguating a token based
on what came immediately before it (JS's `/` being a regex-literal
start or a division operator, depending on the preceding token).
- [Ichor.Toolkit.Codegen](Ichor.Toolkit.Codegen.md): Small, IR-agnostic `quote`/`unquote` mechanics -- not a code
generator, and deliberately not a framework for building one (no
tree-walker, no visitor dispatch, no assumed node shape, no assumed
function-signature convention). Each function here solves exactly one
mechanical pain point that recurs when hand-writing a compiler-to-
quoted-Elixir backend, extracted from `Grammar.Native`'s own codegen
(`CharCompiler`, `RuleCompiler`, `Native.LR`) after the same handful
of tricks turned up independently duplicated across (and within) those
modules -- the same duplication signal that justified
`Ichor.Toolkit.Fixpoint`.

- [Ichor.Toolkit.Fixpoint](Ichor.Toolkit.Fixpoint.md): The generic "repeatedly recompute a value from itself until it stops
changing" primitive -- extracted from a pattern that had already been
hand-written three separate times in Ichor's own compiler internals
before this existed: `Grammar.Analysis`'s private `fixpoint/3`
(nullable-set and always-empty-set computation, both over a
`MapSet.t(atom())` of rule/token names) and `Grammar.LRTable.Sets`'
private `fixpoint/2`/`fixpoint_map/2` (FIRST/FOLLOW-set computation,
one over a `MapSet`, one over a `Map`). None of those three needed
anything MapSet- or Map-specific about the *iteration* itself -- only
about what `step_fn` does with the value in between -- which is
exactly what this generalizes: `step_fn` closes over whatever fixed
data it needs (a grammar's own rule list, a graph, ...), this module
never sees that data at all, only the value being iterated.
- [Ichor.Toolkit.Graph](Ichor.Toolkit.Graph.md): Small, generic graph primitives -- extracted from `Grammar.Analysis`'s
own private `reachable/2`/`do_reachable/3` (used to detect left
recursion: a rule is left-recursive exactly when it's reachable from
its own leading references). Nothing here is grammar-specific --
`neighbors_fn` is the caller's own adjacency, a plain function rather
than a hardcoded map, so this works over whatever graph an author's
own AST/IR/dependency structure represents, not just Ichor's own
rule-reference graphs.

- [Ichor.Toolkit.Layout](Ichor.Toolkit.Layout.md): The off-side-rule (indentation-sensitive layout) algorithm Python's
own tokenizer uses, and Haskell's/F#'s: given each logical line's
indentation width, in order, maintain a stack of open indentation
levels and emit `:indent`/`:dedent` markers as the width rises or
falls -- so a parser downstream never has to know about columns at
all, just consumes ordinary tokens.
- [Ichor.Toolkit.Pratt](Ichor.Toolkit.Pratt.md): Precedence-climbing (Pratt) expression parsing over a runtime-mutable
operator table -- extracted from a pattern already hand-rolled once,
`OpExprTest.Operators.climb/6` (`test/support/opexpr_operators.ex`),
the worked example proving `Grammar.IR.Custom`/`@native(...)` rule-
position dispatch works at all. That fixture only ever needed infix
chains; this generalizes to prefix and postfix too, since those are
exactly what Track 1's own motivating scenarios need (Prolog's
`op/3` supports `fy`/`fx` prefix and `xf`/`yf` postfix operators
alongside infix ones, Haskell fixity declarations are infix-only but
mixfix notation is not).
- [Ichor.Toolkit.Result](Ichor.Toolkit.Result.md): The generic "walk a collection, threading an accumulator through a
fallible step, stopping at the first failure" primitive -- extracted
from a pattern independently hand-rolled at least ten times across
Ichor's own compiler internals before this existed: `build_ruleset/1`
(duplicated near-verbatim across `Ichor.EBNF.ISO.Actions`,
`Ichor.EBNF.W3C.Actions`, `Ichor.BNF.Actions`, `Ichor.ABNF.Actions`, and
`Ichor.PEG.Actions`), `Ichor.Backtrack.Bindings`' own `unify_compound/5`
inner reduce over zipped compound-term arguments (whose failure
sentinel is a bare `:fail`, not `{:error, _}`), `Aether.Eval`'s own
`process_defs/2`, and -- byte-for-byte identical to each other --
`Ichor.Actions`' own `eval_one/2` list-of-captures branch, and
`Aether.Eval`'s own `convert_list/3` and `convert_seq_terms/3`.
- [Ichor.Toolkit.Scope](Ichor.Toolkit.Scope.md): A generic nested lexical scope / symbol table: a stack of name->value
bindings, innermost scope first, supporting the "define here, look up
through enclosing scopes" shape almost every real language's semantic
analysis needs (variable resolution, nested function/block scoping,
...). No existing internal precedent to extract this from (unlike
`Ichor.Toolkit.Fixpoint`/`Graph`) -- a from-scratch design, but a
standard, well-understood one.
- [Ichor.Toolkit.TermWalk](Ichor.Toolkit.TermWalk.md): Generic recursion over any `Ichor.Backtrack.Term` implementation --
extracted from a pattern independently duplicated inside
`Ichor.Toolkit.TypeScheme` itself: `resolve_deep/3` and the private
`substitute/3` shared the identical "apply a transform, then (if the
result is compound) deconstruct, recurse into every argument with the
same transform, and reconstruct" recursive skeleton, differing only in
what the transform itself does at each step (chase a variable's
binding one level, or substitute a specific variable for a fresh one);
`free_vars/3` is the same recursive shape again, but folding into an
accumulator instead of rebuilding a term.
- [Ichor.Toolkit.TypeScheme](Ichor.Toolkit.TypeScheme.md): Hindley-Milner-style let-polymorphism: `generalize/4` and
`instantiate/3`, built on `Ichor.Backtrack.Bindings.unify_occurs_check/4`
(plain `unify/4` deliberately has no occurs-check, matching ISO
Prolog's own default -- a type checker needs one, or unifying a type
variable with a type containing itself would silently build an
infinite type instead of failing).

- Core
  - [Ichor](Ichor.md): Ichor reads grammar definitions -- its own language Aether, plus ABNF,
BNF, EBNF, and PEG importers -- and turns each one into a working
Lexer + Parser + Executor for whatever language the grammar describes,
via either an interpreted VM backend (`Grammar.VM`) or compile-time
native codegen (`__using__/1` below).
  - [Ichor.Actions](Ichor.Actions.md): How a parsed Aether AST turns into an actual result -- a sandboxed
program's final value, a config's map/struct, a query's result set, or
transpiled output. One mechanism, used differently per grammar: a
grammar's Actions module only implements the rules/tokens it needs
custom behavior for (`@optional_callbacks`); everything else falls back
to the defaults below.
  - [Ichor.Capture](Ichor.Capture.md): Every capture an action receives bundles the raw parsed structure with
a callable to evaluate it: `node` is always available, unevaluated --
what `quote` returns directly. `eval` is what an action calls when it
actually wants a child's value, with an explicit context it controls.
Never evaluated automatically -- that's the whole point: a special
form like `if` or `quote` decides *whether* and *when* to call `eval`
at all, which is exactly what lets an untaken `if` branch, or `quote`'s
own argument, go unevaluated (and unbound-symbol/nonterminating errors
inside them stay latent) instead of always eagerly running.

  - [Ichor.Error](Ichor.Error.md): One error struct, reused across every stage -- Lexer, Parser, analysis
pass, and `Ichor.Actions` -- so errors look the same regardless of
origin.
  - [Ichor.Node](Ichor.Node.md): The default-fallback shape `Ichor.Actions` builds for a matched rule
that has more than one meaningful capture (a rule with exactly one
capture passes that capture's own value straight through instead):
`rule` names the matched rule, `captures` maps each capture name to its
already-evaluated value (or a list of values, for a name captured more
than once -- e.g. inside a `*`), and `span` locates the whole match
back in the original input.

- Grammar IR
  - [Grammar.IR](Grammar.IR.md): The normalized grammar AST every Ichor front-end (Aether, ABNF, BNF,
EBNF, PEG) compiles down to, and that every later stage -- the analysis
pass, `Grammar.VM`, native codegen -- consumes instead of talking to any
particular front-end's own syntax.
  - [Grammar.IR.AndPred](Grammar.IR.AndPred.md): Positive lookahead, consumes nothing: `&expr`.
  - [Grammar.IR.Any](Grammar.IR.Any.md): Matches any single character: `.`.
  - [Grammar.IR.Capture](Grammar.IR.Capture.md): Named capture for AST construction: `name:expr`.
  - [Grammar.IR.CharClass](Grammar.IR.CharClass.md): Character class, including unicode ranges: `[a-z0-9_]`.
  - [Grammar.IR.Choice](Grammar.IR.Choice.md): Ordered PEG choice: `a | b | c` -- first match wins, always.
  - [Grammar.IR.Indent](Grammar.IR.Indent.md): Indentation-sensitive block: `@indent(expr)` / `@samecol(expr)`.
  - [Grammar.IR.Literal](Grammar.IR.Literal.md): Exact string match.
  - [Grammar.IR.Meta](Grammar.IR.Meta.md): Source-location metadata carried by every `Grammar.IR` node, so an error
raised at any later stage (analysis, either backend) can still point
back at the exact line/column in the original grammar file rather than
just naming an IR node in the abstract.

  - [Grammar.IR.NotPred](Grammar.IR.NotPred.md): Negative lookahead, consumes nothing: `!expr`.
  - [Grammar.IR.Opt](Grammar.IR.Opt.md): Zero or one: `expr?`.
  - [Grammar.IR.Plus](Grammar.IR.Plus.md): One or more: `expr+`.
  - [Grammar.IR.Rep](Grammar.IR.Rep.md): Bounded repetition: `expr{n}`, `expr{n,}`, `expr{n,m}`.
  - [Grammar.IR.RuleRef](Grammar.IR.RuleRef.md): Reference to another rule (or token), by name.
  - [Grammar.IR.Seq](Grammar.IR.Seq.md): Ordered sequence of sub-expressions: `a b c`.
  - [Grammar.IR.Star](Grammar.IR.Star.md): Zero or more: `expr*`.

- Aether Front-end
  - [Aether.Grammar](Aether.Grammar.md): The fully compiled output of `Aether.Parser`: grammar-wide settings plus
every token and rule body, each already resolved to `Grammar.IR`.
  - [Aether.Lexer](Aether.Lexer.md): Turns `.aether` source text into a flat list of `Aether.Token`s.
  - [Aether.Parser](Aether.Parser.md): Parses `.aether` source into a fully resolved `Aether.Grammar`, in two
stages: `Aether.Reader` (pure syntax -- a token/rule reference split,
pragmas, the expression grammar -- into a concrete syntax tree with no
desugaring) followed by `Aether.Eval` (that CST into `Grammar.IR`,
handling everything that needs whole-grammar context: predefined-token
override/use tracking, case-insensitivity, character-class/regex
desugaring, inline-literal promotion, `@skip` splicing, and final
`@root`/`@skip` validation).
  - [Aether.Token](Aether.Token.md): A single lexical token produced by `Aether.Lexer` from `.aether` source
text.

- Analysis
  - [Grammar.Analysis](Grammar.Analysis.md): The analysis pass between a front-end (`Aether.Parser`, or one of the
ABNF/BNF/EBNF/PEG importers) and either backend (`Grammar.VM` or native
codegen)

- VM Backend
  - [Grammar.VM](Grammar.VM.md): The interpreted runtime backend: compiles an `Aether.Grammar` to
bytecode and runs it against real input.
  - [Grammar.VM.CharCompiler](Grammar.VM.CharCompiler.md): Compiles every token in a grammar (character-level: `Literal`,
`CharClass`, `Any`, and `RuleRef` to another token -- the only node
shapes an *ordinary* token body can contain) into one linked
`Grammar.VM.Program`, run by `Grammar.VM.Tokenizer`.
  - [Grammar.VM.CharInterpreter](Grammar.VM.CharInterpreter.md): Runs a `Grammar.VM.CharCompiler`-produced program against raw input
text -- the machine `Grammar.VM.Tokenizer` uses to test one token at a
given starting position.
  - [Grammar.VM.Compiler](Grammar.VM.Compiler.md): The combinator half of Grammar.VM's bytecode compilation, shared
between the character-level compiler (token bodies, run by the lexer)
and the token-stream-level compiler (rule bodies, run by the parser).
`Seq`/`Choice`/`Star`/`Plus`/`Opt`/`Rep`/`AndPred`/`NotPred` compile
identically either way -- only the leaf nodes differ
(`Literal`/`CharClass`/`Any`/`RuleRef` for tokens; `RuleRef`/`Indent`/
`Capture` for rules -- tokens can never contain a `Capture`), so each
caller supplies its own `leaf_fun` for those.
  - [Grammar.VM.Linker](Grammar.VM.Linker.md): Turns `[{name, ops}]` -- one label-relative op list per token or rule,
as produced by `Grammar.VM.Compiler` -- into a single linked
`Grammar.VM.Program`.
  - [Grammar.VM.Program](Grammar.VM.Program.md): Linked PEG bytecode: every token (or every rule) in a grammar, compiled
into one shared instruction tuple so `:call` can jump between them by
index, plus a name -> entry-index map for looking up where to start.
  - [Grammar.VM.RuleCompiler](Grammar.VM.RuleCompiler.md): Compiles every rule in a grammar into one linked `Grammar.VM.Program`,
run by `Grammar.VM.TokenInterpreter` against the lexer's token stream
(never against raw characters -- Aether's two-stage Lexer -> Parser
split). A `RuleRef` compiles to `{:token, name}` (consume one stream
token of that type) when it names a token, or `{:call, name}` (jump
into that rule's own compiled code) when it names another rule -- the
two are otherwise indistinguishable in the IR, so the grammar's own
token/rule namespaces are what disambiguate.
  - [Grammar.VM.Token](Grammar.VM.Token.md): One lexed token from `Grammar.VM.Tokenizer` -- the target grammar's own
tokens (e.g. a calculator's `NUMBER`, `"+"`), not to be confused with
`Aether.Token` (Ichor's front-end lexing `.aether` source itself).
  - [Grammar.VM.TokenInterpreter](Grammar.VM.TokenInterpreter.md): Runs a `Grammar.VM.RuleCompiler`-produced program against a lexed token
stream -- the machine `Grammar.VM` uses for the parser stage.

- Native Backend
  - [Grammar.Native](Grammar.Native.md): The compile-time codegen backend: turns a validated `%Aether.Grammar{}`
into quoted Elixir function definitions -- the same two-stage Lexer ->
Parser split `Grammar.VM` compiles to bytecode for, but here as direct
function calls a `use Ichor, grammar:, actions:` caller splices
straight into its own module (via `Ichor`'s `__using__` macro),
skipping bytecode interpretation entirely.
  - [Grammar.Native.CharCompiler](Grammar.Native.CharCompiler.md): Compiles every token in a grammar (character-level: `Literal`,
`CharClass`, `Any`, `RuleRef` to another token, and the combinators
`Seq`/`Choice`/`Star`/`Plus`/`Opt`/`Rep`/`AndPred`/`NotPred` -- the
same shapes `Grammar.VM.CharCompiler` compiles) into quoted Elixir
function definitions instead of bytecode.
  - [Grammar.Native.RuleCompiler](Grammar.Native.RuleCompiler.md): Compiles every rule in a grammar into quoted Elixir function
definitions run against the lexer's token stream (never raw
characters -- Aether's own two-stage split), mirroring
`Grammar.VM.RuleCompiler`'s own semantics exactly (bare-reference
implicit self-capture, `Indent`/`@samecol`, capture-shape rules) but
producing direct function calls instead of bytecode.

- Tooling
  - [Grammar.Tokens](Grammar.Tokens.md): Token introspection for a compiled `Aether.Grammar` -- the logic behind
`mix ichor.tokens`. Depends only on the Aether front-end's output
(`token_order`/`tokens`/`anon_tokens`), not the analysis pass or either
backend, so it works on any grammar that parses, even one the analysis
pass would otherwise reject.

- Format Importers
  - [Ichor.ABNF](Ichor.ABNF.md): The ABNF (RFC 5234 + RFC 7405) front-end: `priv/grammar/abnf.aether`
parsed by Aether's own front-end, compiled by the native codegen
backend (`use Ichor`), dispatching to `Ichor.ABNF.Actions`
(`Grammar.IR` as its target category, same as `Regex.Actions`).
  - [Ichor.ABNF.Actions](Ichor.ABNF.Actions.md): Turns a parsed ABNF `rulelist` (RFC 5234 + RFC 7405) into a real
`%{rule_name_atom => Grammar.IR.expr()}` map -- one `Grammar.IR` tree
per ABNF rule, mirroring how `Regex.Actions` turns a single
`/pattern/` into one `Grammar.IR` tree. This is an importer's own
output, not a runnable `Aether.Grammar` -- ABNF has no lexer/rule
(token/parser) split the way Aether does, so deciding which of an
ABNF ruleset's productions become Aether tokens vs. rules is a
separate concern this module doesn't address.
  - [Ichor.BNF](Ichor.BNF.md): The classical BNF front-end -- no single citable standard exists;
this follows the ALGOL 60 Report's own convention with quoted
terminals, matching how BNF is actually written today (see
`Ichor.BNF.Actions`'s own moduledoc for the deliberate deviation that
is): `priv/grammar/bnf.aether` parsed by Aether's own front-end,
compiled by the native codegen backend (`use Ichor`), dispatching to
`Ichor.BNF.Actions` (`Grammar.IR` as its target category).
  - [Ichor.BNF.Actions](Ichor.BNF.Actions.md): Turns a parsed classical BNF `grammar_file` into a real
`%{nonterminal_name_atom => Grammar.IR.expr()}` map -- one
`Grammar.IR` tree per rule, the same target category `Ichor.ABNF.Actions`
and `Regex.Actions` both use.
  - [Ichor.EBNF.ISO](Ichor.EBNF.ISO.md): The ISO/IEC 14977 EBNF front-end: `priv/grammar/ebnf-iso.aether`
parsed by Aether's own front-end, compiled by the native codegen
backend (`use Ichor`), dispatching to `Ichor.EBNF.ISO.Actions`
(`Grammar.IR` as its target category).
  - [Ichor.EBNF.ISO.Actions](Ichor.EBNF.ISO.Actions.md): Turns a parsed ISO/IEC 14977 EBNF `syntax` into a real
`%{meta_identifier_atom => Grammar.IR.expr()}` map -- one `Grammar.IR`
tree per rule, the same target category `Ichor.ABNF.Actions` and
`Ichor.BNF.Actions` both use.
  - [Ichor.EBNF.W3C](Ichor.EBNF.W3C.md): The W3C-style EBNF front-end -- the notation the XML 1.0 spec's own
section 6 "Notation" uses, also shared by XQuery/XPath's grammars:
`priv/grammar/ebnf-w3c.aether` parsed by Aether's own front-end,
compiled by the native codegen backend (`use Ichor`), dispatching to
`Ichor.EBNF.W3C.Actions` (`Grammar.IR` as its target category).
  - [Ichor.EBNF.W3C.Actions](Ichor.EBNF.W3C.Actions.md): Turns a parsed W3C-style EBNF `grammar_file` (the notation the XML 1.0
spec's own section 6 uses, also shared by XQuery/XPath's grammars)
into a real `%{ident_atom => Grammar.IR.expr()}` map -- one
`Grammar.IR` tree per rule, the same target category every other
importer this project builds uses.
  - [Ichor.PEG](Ichor.PEG.md): The PEG front-end (Ford's paper, pest/PEG.js-style convention):
`priv/grammar/peg.aether` parsed by Aether's own front-end, compiled
by the native codegen backend (`use Ichor`), dispatching to
`Ichor.PEG.Actions` (`Grammar.IR` as its target category).
  - [Ichor.PEG.Actions](Ichor.PEG.Actions.md): Turns a parsed PEG `grammar_file` (Ford's paper, pest/PEG.js-style
convention) into a real `%{ident_atom => Grammar.IR.expr()}` map --
one `Grammar.IR` tree per rule, the same target category every other
importer this project builds uses.

## Mix Tasks

- Tooling
  - [mix ichor.tokens](Mix.Tasks.Ichor.Tokens.md): Reads a `.aether` grammar file and lists every token it defines --
user-declared, anonymous (auto-promoted from an inline rule literal),
and the five predefined tokens (always present whether overridden or
left at their default) -- in `token_order`, the same order the Lexer's
maximal-munch tie-break actually uses, alongside a rendered pattern for
each (`Grammar.Tokens`).

