API Reference Ichor v#0.2.0

The semantic half of Aether's front-end: turns an Aether.Reader.Grammar CST into a fully resolved Aether.Grammar, with every body already Grammar.IR.

The fully compiled output of Aether.Parser: grammar-wide settings plus every token and rule body, each already resolved to Grammar.IR.

Turns .aether source text into a flat list of Aether.Tokens.

Parses .aether source into a fully resolved Aether.Grammar, in two stages: Aether.Reader (pure syntax -- a token/rule reference split, pragmas, the expression grammar -- into a concrete syntax tree with no desugaring) followed by Aether.Eval (that CST into Grammar.IR, handling everything that needs whole-grammar context: predefined-token override/use tracking, case-insensitivity, character-class/regex desugaring, inline-literal promotion, @skip splicing, and final @root/@skip validation).

The pure-syntax half of Aether's front-end: turns .aether source (via Aether.Lexer) into a concrete syntax tree that's a faithful record of what was written -- every pragma, token, and rule definition, in file order -- with no desugaring and no cross-definition semantics applied.

The CST Aether.Reader.read/2 produces: grammar-wide pragmas plus every token/rule definition in file order, each body an Aether.Reader.cst() tree. Aether.Eval.build/1 is the only consumer.

A single lexical token produced by Aether.Lexer from .aether source text.

The analysis pass between a front-end (Aether.Parser, or one of the ABNF/BNF/EBNF/PEG importers) and either backend (Grammar.VM or native codegen)

The interpreted graph-structured-stack backend: runs the same Grammar.LRTable SLR(1) table Grammar.LR does, but accepts conflicts instead of requiring their absence -- every action in a conflict cell is taken, forking the parse across a real Grammar.GLR.GSS (node-sharing, not independent per-branch stacks: two derivations that reach the same state at the same position merge, which is what keeps this from blowing up combinatorially on a grammar with only a handful of local conflicts). The GSS-driving loop itself lives in Grammar.GLR.Runtime, shared unchanged with Grammar.Native.GLR -- this module's own job is just building the table (recompiled on every call, the same "VM interpreted" convention Grammar.VM and Grammar.LR already follow) and wrapping its plain action/goto maps into the closures Runtime.run/6 expects.

The normalized grammar AST every Ichor front-end (Aether, ABNF, BNF, EBNF, PEG) compiles down to, and that every later stage -- the analysis pass, Grammar.VM, native codegen -- consumes instead of talking to any particular front-end's own syntax.

Positive lookahead, consumes nothing: &expr.

Matches any single character: ..

Named capture for AST construction: name:expr.

Character class, including unicode ranges: [a-z0-9_].

Ordered PEG choice: a | b | c -- first match wins, always.

@native("Module", "function", dep1, dep2, ...): a rule body that delegates to hand-written Elixir code (an Ichor.CustomRule implementation) instead of ordinary Grammar.IR combinators -- the escape hatch for constructs no static PEG grammar can express (a Prolog-style mutable operator-precedence table, C's typedef-vs- expression ambiguity, and similar "what matches here depends on something read earlier" problems).

@native("Module", "function", dep1, ...) at token position: a token whose entire body is hand-written Elixir code (an Ichor.CustomLexeme implementation) instead of ordinary character-level combinators -- the escape hatch for lexical constructs no fixed maximal-munch tokenizer can express (a heredoc's dynamic terminator, a string literal with embedded interpolated expressions).

Indentation-sensitive block: @indent(expr) / @samecol(expr).

Exact string match.

Source-location metadata carried by every Grammar.IR node, so an error raised at any later stage (analysis, either backend) can still point back at the exact line/column in the original grammar file rather than just naming an IR node in the abstract.

Negative lookahead, consumes nothing: !expr.

Zero or one: expr?.

One or more: expr+.

Bounded repetition: expr{n}, expr{n,}, expr{n,m}.

Reference to another rule (or token), by name.

Ordered sequence of sub-expressions: a b c.

Zero or more: expr*.

The deterministic bottom-up backend: builds an SLR(1) table via Grammar.LRTable and requires it be conflict-free -- a real conflict is a compile-time error here (compile/1), never something forked through (that's what Grammar.GLR is for). No graph-structured-stack bookkeeping at all: an ordinary shift-reduce loop over a single stack.

Canonical LR(0) item-set construction (closure/goto over Grammar.LRTable.Desugar's flat production list), plus SLR(1) action/goto table construction on top of it.

Builds an SLR(1) action/goto table (Grammar.LRTable, from ichor_runtime) from an @engine lr/@engine glr grammar -- shared by both Grammar.LR (which additionally requires the result be conflict-free) and Grammar.GLR (which accepts conflicts and forks over them at runtime).

Flattens grammar.rules' PEG-shaped IR into the flat CFG production list Grammar.LRTable.Production describes, for Grammar.LRTable's LR(0)/SLR(1) construction to consume.

Standard textbook nullable/FIRST/FOLLOW fixpoint algorithms, over Grammar.LRTable.Desugar's flat CFG production list rather than the original PEG IR -- recomputed fresh (not reusing Grammar.Analysis's own compute_nullable/1) since the augmented nonterminal set here includes every helper nonterminal a Star/Plus/Opt/Rep/group desugared into, which Grammar.Analysis never sees.

The compile-time codegen backend: turns a validated %Aether.Grammar{} into quoted Elixir function definitions -- the same two-stage Lexer -> Parser split Grammar.VM compiles to bytecode for, but here as direct function calls a use Ichor, grammar:, actions: caller splices straight into its own module (via Ichor's __using__ macro), skipping bytecode interpretation entirely.

Compiles every token in a grammar (character-level: Literal, CharClass, Any, RuleRef to another token, and the combinators Seq/Choice/Star/Plus/Opt/Rep/AndPred/NotPred -- the same shapes Grammar.VM.CharCompiler compiles) into quoted Elixir function definitions instead of bytecode.

The compile-time codegen backend for @engine glr grammars: builds the SLR(1) table via Grammar.LRTable once, at Elixir-compile-time (conflicts are expected here, never rejected -- that's Grammar.Native.LR's job), and compiles the action/goto lookup into per-state generated function clauses instead of the interpreted Grammar.GLR's Map.gets.

The compile-time codegen backend for @engine lr grammars: builds the SLR(1) table via Grammar.LRTable once, at Elixir-compile-time (inside use Ichor's macro expansion), and requires it be conflict-free -- same requirement Grammar.LR.compile/1 enforces at runtime, just checked once here instead of on every call.

Compiles every rule in a grammar into quoted Elixir function definitions run against the lexer's token stream (never raw characters -- Aether's own two-stage split), mirroring Grammar.VM.RuleCompiler's own semantics exactly (bare-reference implicit self-capture, Indent/@samecol, capture-shape rules) but producing direct function calls instead of bytecode.

Generates the Tokenizer -> Lexer half of a compiled grammar's tokenize/2 (char-level maximal munch via Grammar.Native.CharCompiler, then @keywords/@refine reclassification via the shared, backend-agnostic Grammar.Lexer) -- extracted once out of Grammar.Native.generate/2 so Grammar.Native.LR/Grammar.Native.GLR can reuse it unchanged: every engine consumes the exact same token stream, only the parser half differs (direct PEG combinator calls, vs. compiled LR/GLR state dispatch).

Token introspection for a compiled Aether.Grammar -- the logic behind mix ichor.tokens. Depends only on the Aether front-end's output (token_order/tokens/anon_tokens), not the analysis pass or either backend, so it works on any grammar that parses, even one the analysis pass would otherwise reject.

The interpreted runtime backend: compiles an Aether.Grammar to bytecode and runs it against real input.

Compiles every token in a grammar (character-level: Literal, CharClass, Any, and RuleRef to another token -- the only node shapes an ordinary token body can contain) into one linked Grammar.VM.Program, run by Grammar.VM.Tokenizer.

Runs a Grammar.VM.CharCompiler-produced program against raw input text -- the machine Grammar.VM.Tokenizer uses to test one token at a given starting position.

The combinator half of Grammar.VM's bytecode compilation, shared between the character-level compiler (token bodies, run by the lexer) and the token-stream-level compiler (rule bodies, run by the parser). Seq/Choice/Star/Plus/Opt/Rep/AndPred/NotPred compile identically either way -- only the leaf nodes differ (Literal/CharClass/Any/RuleRef for tokens; RuleRef/Indent/ Capture for rules -- tokens can never contain a Capture), so each caller supplies its own leaf_fun for those.

Turns [{name, ops}] -- one label-relative op list per token or rule, as produced by Grammar.VM.Compiler -- into a single linked Grammar.VM.Program.

Linked PEG bytecode: every token (or every rule) in a grammar, compiled into one shared instruction tuple so :call can jump between them by index, plus a name -> entry-index map for looking up where to start.

Compiles every rule in a grammar into one linked Grammar.VM.Program, run by Grammar.VM.TokenInterpreter against the lexer's token stream (never against raw characters -- Aether's two-stage Lexer -> Parser split). A RuleRef compiles to {:token, name} (consume one stream token of that type) when it names a token, or {:call, name} (jump into that rule's own compiled code) when it names another rule -- the two are otherwise indistinguishable in the IR, so the grammar's own token/rule namespaces are what disambiguate.

Runs a Grammar.VM.RuleCompiler-produced program against a lexed token stream -- the machine Grammar.VM uses for the parser stage.

Tokenizes a full input string against a Grammar.VM.CharCompiler program, using maximal munch: at each position, every declared token is tried, the longest match wins, ties are broken by declaration order. Trivia/skip tokens are emitted like any other -- filtering them out is the parser's job, via the Star(skip_token) that Aether.Parser already spliced into rule bodies, not the lexer's.

Ichor reads grammar definitions -- its own language Aether, plus ABNF, BNF, EBNF, and PEG importers -- and turns each one into a working Lexer + Parser + Executor for whatever language the grammar describes, via either an interpreted VM backend (Grammar.VM) or compile-time native codegen (__using__/1 below).

The ABNF (RFC 5234 + RFC 7405) front-end: priv/grammar/abnf.aether parsed by Aether's own front-end, compiled by the native codegen backend (use Ichor), dispatching to Ichor.ABNF.Actions (Grammar.IR as its target category, same as Regex.Actions).

Turns a parsed ABNF rulelist (RFC 5234 + RFC 7405) into a real %{rule_name_atom => Grammar.IR.expr()} map -- one Grammar.IR tree per ABNF rule, mirroring how Regex.Actions turns a single /pattern/ into one Grammar.IR tree. This is an importer's own output, not a runnable Aether.Grammar -- ABNF has no lexer/rule (token/parser) split the way Aether does, so deciding which of an ABNF ruleset's productions become Aether tokens vs. rules is a separate concern this module doesn't address.

The classical BNF front-end -- no single citable standard exists; this follows the ALGOL 60 Report's own convention with quoted terminals, matching how BNF is actually written today (see Ichor.BNF.Actions's own moduledoc for the deliberate deviation that is): priv/grammar/bnf.aether parsed by Aether's own front-end, compiled by the native codegen backend (use Ichor), dispatching to Ichor.BNF.Actions (Grammar.IR as its target category).

Turns a parsed classical BNF grammar_file into a real %{nonterminal_name_atom => Grammar.IR.expr()} map -- one Grammar.IR tree per rule, the same target category Ichor.ABNF.Actions and Regex.Actions both use.

The ISO/IEC 14977 EBNF front-end: priv/grammar/ebnf-iso.aether parsed by Aether's own front-end, compiled by the native codegen backend (use Ichor), dispatching to Ichor.EBNF.ISO.Actions (Grammar.IR as its target category).

Turns a parsed ISO/IEC 14977 EBNF syntax into a real %{meta_identifier_atom => Grammar.IR.expr()} map -- one Grammar.IR tree per rule, the same target category Ichor.ABNF.Actions and Ichor.BNF.Actions both use.

The W3C-style EBNF front-end -- the notation the XML 1.0 spec's own section 6 "Notation" uses, also shared by XQuery/XPath's grammars: priv/grammar/ebnf-w3c.aether parsed by Aether's own front-end, compiled by the native codegen backend (use Ichor), dispatching to Ichor.EBNF.W3C.Actions (Grammar.IR as its target category).

Turns a parsed W3C-style EBNF grammar_file (the notation the XML 1.0 spec's own section 6 uses, also shared by XQuery/XPath's grammars) into a real %{ident_atom => Grammar.IR.expr()} map -- one Grammar.IR tree per rule, the same target category every other importer this project builds uses.

Turns an ABNF/BNF/ISO-EBNF/PEG ruleset (as Ichor.ABNF/Ichor.BNF/ Ichor.EBNF.ISO/Ichor.PEG produce them -- %{rule_name_atom => Grammar.IR.expr()}, explicitly documented as "not a runnable Aether.Grammar") into one that actually is: picks a root (the source's own first-declared rule, unless overridden), assembles a full %Aether.Grammar{}, and (for ABNF specifically) fills in RFC 5234 Appendix B's implicit "core rules" (ALPHA, DIGIT, CRLF, ...) for anything the source references but never defines itself -- extremely common in real ABNF, which generally assumes those are always available.

The PEG front-end (Ford's paper, pest/PEG.js-style convention): priv/grammar/peg.aether parsed by Aether's own front-end, compiled by the native codegen backend (use Ichor), dispatching to Ichor.PEG.Actions (Grammar.IR as its target category).

Turns a parsed PEG grammar_file (Ford's paper, pest/PEG.js-style convention) into a real %{ident_atom => Grammar.IR.expr()} map -- one Grammar.IR tree per rule, the same target category every other importer this project builds uses.

The behaviour a @refine("Module", "function", ...) token suffix dispatches to -- reclassifying (or validating/decoding) a token the Tokenizer already matched, before the Parser ever sees it. @keywords is sugar for the common table-lookup case of this same mechanism (see Grammar.Lexer); @refine is the escape hatch for anything needing real logic: escape-sequence decoding, or disambiguating a token based on what came immediately before it (JS's / being a regex-literal start or a division operator, depending on the preceding token).

Small, IR-agnostic quote/unquote mechanics -- not a code generator, and deliberately not a framework for building one (no tree-walker, no visitor dispatch, no assumed node shape, no assumed function-signature convention). Each function here solves exactly one mechanical pain point that recurs when hand-writing a compiler-to- quoted-Elixir backend, extracted from Grammar.Native's own codegen (CharCompiler, RuleCompiler, Native.LR) after the same handful of tricks turned up independently duplicated across (and within) those modules -- the same duplication signal that justified Ichor.Toolkit.Fixpoint.

The generic "repeatedly recompute a value from itself until it stops changing" primitive -- extracted from a pattern that had already been hand-written three separate times in Ichor's own compiler internals before this existed: Grammar.Analysis's private fixpoint/3 (nullable-set and always-empty-set computation, both over a MapSet.t(atom()) of rule/token names) and Grammar.LRTable.Sets' private fixpoint/2/fixpoint_map/2 (FIRST/FOLLOW-set computation, one over a MapSet, one over a Map). None of those three needed anything MapSet- or Map-specific about the iteration itself -- only about what step_fn does with the value in between -- which is exactly what this generalizes: step_fn closes over whatever fixed data it needs (a grammar's own rule list, a graph, ...), this module never sees that data at all, only the value being iterated.

Small, generic graph primitives -- extracted from Grammar.Analysis's own private reachable/2/do_reachable/3 (used to detect left recursion: a rule is left-recursive exactly when it's reachable from its own leading references). Nothing here is grammar-specific -- neighbors_fn is the caller's own adjacency, a plain function rather than a hardcoded map, so this works over whatever graph an author's own AST/IR/dependency structure represents, not just Ichor's own rule-reference graphs.

The off-side-rule (indentation-sensitive layout) algorithm Python's own tokenizer uses, and Haskell's/F#'s: given each logical line's indentation width, in order, maintain a stack of open indentation levels and emit :indent/:dedent markers as the width rises or falls -- so a parser downstream never has to know about columns at all, just consumes ordinary tokens.

A generic nested lexical scope / symbol table: a stack of name->value bindings, innermost scope first, supporting the "define here, look up through enclosing scopes" shape almost every real language's semantic analysis needs (variable resolution, nested function/block scoping, ...). No existing internal precedent to extract this from (unlike Ichor.Toolkit.Fixpoint/Graph) -- a from-scratch design, but a standard, well-understood one.

Hindley-Milner-style let-polymorphism: generalize/4 and instantiate/3, built on Ichor.Backtrack.Bindings.unify_occurs_check/4 (plain unify/4 deliberately has no occurs-check, matching ISO Prolog's own default -- a type checker needs one, or unifying a type variable with a type containing itself would silently build an infinite type instead of failing).

Runs the exact same parse -> analyze -> native-codegen pipeline as use Ichor, grammar: ..., actions: ... (Ichor.generate/3), but once, from the command line, writing the result to disk as an ordinary module instead of splicing it into a macro expansion.

Reads a .aether grammar file and lists every token it defines -- user-declared, anonymous (auto-promoted from an inline rule literal), and the five predefined tokens (always present whether overridden or left at their default) -- in token_order, the same order the Lexer's maximal-munch tie-break actually uses, alongside a rendered pattern for each (Grammar.Tokens).