API Reference Ichor v#0.7.2

The semantic half of Aether's front-end: turns an Aether.Reader.Grammar CST into a fully resolved Aether.Grammar, with every body already Grammar.IR.

The fully compiled output of Aether.Parser: grammar-wide settings plus every token and rule body, each already resolved to Grammar.IR.

Turns .aether source text into a flat list of Aether.Tokens.

Parses .aether source into a fully resolved Aether.Grammar, in two stages: Aether.Reader (pure syntax -- a token/rule reference split, pragmas, the expression grammar -- into a concrete syntax tree with no desugaring) followed by Aether.Eval (that CST into Grammar.IR, handling everything that needs whole-grammar context: predefined-token override/use tracking, case-insensitivity, character-class/regex desugaring, inline-literal promotion, @skip splicing, and final @root/@skip validation).

The pure-syntax half of Aether's front-end: turns .aether source (via Aether.Lexer) into a concrete syntax tree that's a faithful record of what was written -- every pragma, token, and rule definition, in file order -- with no desugaring and no cross-definition semantics applied.

The CST Aether.Reader.read/2 produces: grammar-wide pragmas plus every token/rule definition in file order, each body an Aether.Reader.cst() tree. Aether.Eval.build/1 is the only consumer.

A single lexical token produced by Aether.Lexer from .aether source text.

The analysis pass between a front-end (Aether.Parser, or one of the ABNF/BNF/EBNF/PEG importers) and either backend (Grammar.VM or native codegen)

The interpreted graph-structured-stack backend: runs the same Grammar.LRTable SLR(1) table Grammar.LR does, but accepts conflicts instead of requiring their absence -- every action in a conflict cell is taken, forking the parse across a real Grammar.GLR.GSS (node-sharing, not independent per-branch stacks: two derivations that reach the same state at the same position merge, which is what keeps this from blowing up combinatorially on a grammar with only a handful of local conflicts). The GSS-driving loop itself lives in Grammar.GLR.Runtime, shared unchanged with Grammar.Native.GLR -- this module's own job is just building the table (recompiled on every call, the same "VM interpreted" convention Grammar.VM and Grammar.LR already follow) and wrapping its plain action/goto maps into the closures Runtime.run/6 expects.

The normalized grammar AST every Ichor front-end (Aether, ABNF, BNF, EBNF, PEG) compiles down to, and that every later stage -- the analysis pass, Grammar.VM, native codegen -- consumes instead of talking to any particular front-end's own syntax.

Positive lookahead, consumes nothing: &expr.

Matches any single character: ..

Named capture for AST construction: name:expr.

Character class, including unicode ranges: [a-z0-9_].

Ordered PEG choice: a | b | c -- first match wins, always.

@native("Module", "function", dep1, dep2, ...): a rule body that delegates to hand-written Elixir code (an Ichor.CustomRule implementation) instead of ordinary Grammar.IR combinators -- the escape hatch for constructs no static PEG grammar can express (a Prolog-style mutable operator-precedence table, C's typedef-vs- expression ambiguity, and similar "what matches here depends on something read earlier" problems).

@native("Module", "function", dep1, ...) at token position: a token whose entire body is hand-written Elixir code (an Ichor.CustomLexeme implementation) instead of ordinary character-level combinators -- the escape hatch for lexical constructs no fixed maximal-munch tokenizer can express (a heredoc's dynamic terminator, a string literal with embedded interpolated expressions).

Indentation-sensitive block: @indent(expr) / @samecol(expr).

Exact string match.

Source-location metadata carried by every Grammar.IR node, so an error raised at any later stage (analysis, either backend) can still point back at the exact line/column in the original grammar file rather than just naming an IR node in the abstract.

Negative lookahead, consumes nothing: !expr.

Zero or one: expr?.

One or more: expr+.

Bounded repetition: expr{n}, expr{n,}, expr{n,m}.

Reference to another rule (or token), by name.

Ordered sequence of sub-expressions: a b c.

Zero or more: expr*.

The deterministic bottom-up backend: builds an SLR(1) table via Grammar.LRTable and requires it be conflict-free -- a real conflict is a compile-time error here (compile/1), never something forked through (that's what Grammar.GLR is for). No graph-structured-stack bookkeeping at all: an ordinary shift-reduce loop over a single stack.

Canonical LR(0) item-set construction (closure/goto over Grammar.LRTable.Desugar's flat production list), plus SLR(1) action/goto table construction on top of it.

Builds an SLR(1) action/goto table (Grammar.LRTable, from ichor_runtime) from an @engine lr/@engine glr grammar -- shared by both Grammar.LR (which additionally requires the result be conflict-free) and Grammar.GLR (which accepts conflicts and forks over them at runtime).

Flattens grammar.rules' PEG-shaped IR into the flat CFG production list Grammar.LRTable.Production describes, for Grammar.LRTable's LR(0)/SLR(1) construction to consume.

Standard textbook nullable/FIRST/FOLLOW fixpoint algorithms, over Grammar.LRTable.Desugar's flat CFG production list rather than the original PEG IR -- recomputed fresh (not reusing Grammar.Analysis's own compute_nullable/1) since the augmented nonterminal set here includes every helper nonterminal a Star/Plus/Opt/Rep/group desugared into, which Grammar.Analysis never sees.

The lossless match of the interpreted shift-reduce backends, Grammar.LR and Grammar.GLR (design/lossless-mode.md): tokenize, run the table with the rules recording, place an error against the input, weave the tree.

The compile-time codegen backend: turns a validated %Aether.Grammar{} into quoted Elixir function definitions -- the same two-stage Lexer -> Parser split Grammar.VM compiles to bytecode for, but here as direct function calls a use Ichor, grammar:, actions: caller splices straight into its own module (via Ichor's __using__ macro), skipping bytecode interpretation entirely.

Compiles every token in a grammar (character-level: Literal, CharClass, Any, RuleRef to another token, and the combinators Seq/Choice/Star/Plus/Opt/Rep/AndPred/NotPred -- the same shapes Grammar.VM.CharCompiler compiles) into quoted Elixir function definitions instead of bytecode.

The compile-time codegen backend for @engine glr grammars: builds the SLR(1) table via Grammar.LRTable once, at Elixir-compile-time (conflicts are expected here, never rejected -- that's Grammar.Native.LR's job), and compiles the action/goto lookup into per-state generated function clauses instead of the interpreted Grammar.GLR's Map.gets.

The compile-time codegen backend for @engine lr grammars: builds the SLR(1) table via Grammar.LRTable once, at Elixir-compile-time (inside use Ichor's macro expansion), and requires it be conflict-free -- same requirement Grammar.LR.compile/1 enforces at runtime, just checked once here instead of on every call.

What lossless: true adds to a generated LR or GLR parser (design/lossless-mode.md): parse_lossless/1,2 and run_lossless/1,2, over a shift-reduce run that records the rules (Grammar.LR.Lossless, or Grammar.GLR.Runtime.run/7 building with Ichor.Syntax.Record.reduced/3).

Compiles every rule in a grammar into quoted Elixir function definitions run against the lexer's token stream (never raw characters -- Aether's own two-stage split), mirroring Grammar.VM.RuleCompiler's own semantics exactly (bare-reference implicit self-capture, Indent/@samecol, capture-shape rules) but producing direct function calls instead of bytecode.

Generates the Tokenizer -> Lexer half of a compiled grammar's tokenize/2 (char-level maximal munch via Grammar.Native.CharCompiler, then @keywords/@refine reclassification via the shared, backend-agnostic Grammar.Lexer) -- extracted once out of Grammar.Native.generate/2 so Grammar.Native.LR/Grammar.Native.GLR can reuse it unchanged: every engine consumes the exact same token stream, only the parser half differs (direct PEG combinator calls, vs. compiled LR/GLR state dispatch).

Token introspection for a compiled Aether.Grammar -- the logic behind mix ichor.tokens. Depends only on the Aether front-end's output (token_order/tokens/anon_tokens), not the analysis pass or either backend, so it works on any grammar that parses, even one the analysis pass would otherwise reject.

The interpreted runtime backend: compiles an Aether.Grammar to bytecode and runs it against real input. No compile-time codegen step -- give it a grammar (from Aether.Parser.parse/2 + Grammar.Analysis.run/1) whenever you have one, including one your own program only learns about at runtime (a user-supplied grammar, a plugin, a REPL's :load). See the tutorial's "Loading a grammar at runtime" section for a full worked example, and "Which path is right for you?" for how this compares to use Ichor/Mix.Tasks.Ichor.Gen.

Compiles every token in a grammar (character-level: Literal, CharClass, Any, and RuleRef to another token -- the only node shapes an ordinary token body can contain) into one linked Grammar.VM.Program, run by Grammar.VM.Tokenizer.

Runs a Grammar.VM.CharCompiler-produced program against raw input text -- the machine Grammar.VM.Tokenizer uses to test one token at a given starting position.

The combinator half of Grammar.VM's bytecode compilation, shared between the character-level compiler (token bodies, run by the lexer) and the token-stream-level compiler (rule bodies, run by the parser). Seq/Choice/Star/Plus/Opt/Rep/AndPred/NotPred compile identically either way -- only the leaf nodes differ (Literal/CharClass/Any/RuleRef for tokens; RuleRef/Indent/ Capture for rules -- tokens can never contain a Capture), so each caller supplies its own leaf_fun for those.

Turns [{name, ops}] -- one label-relative op list per token or rule, as produced by Grammar.VM.Compiler -- into a single linked Grammar.VM.Program.

Linked PEG bytecode: every token (or every rule) in a grammar, compiled into one shared instruction tuple so :call can jump between them by index, plus a name -> entry-index map for looking up where to start.

Compiles every rule in a grammar into one linked Grammar.VM.Program, run by Grammar.VM.TokenInterpreter against the lexer's token stream (never against raw characters -- Aether's two-stage Lexer -> Parser split). A RuleRef compiles to {:token, name} (consume one stream token of that type) when it names a token, or {:call, name} (jump into that rule's own compiled code) when it names another rule -- the two are otherwise indistinguishable in the IR, so the grammar's own token/rule namespaces are what disambiguate.

Runs a Grammar.VM.RuleCompiler-produced program against a lexed token stream -- the machine Grammar.VM uses for the parser stage.

Tokenizes a full input string against a Grammar.VM.CharCompiler program, using maximal munch: at each position, every declared token is tried, the longest match wins, ties are broken by declaration order. Trivia/skip tokens are emitted like any other -- filtering them out is the parser's job, via the Star(skip_token) that Aether.Parser already spliced into rule bodies, not the lexer's.

Ichor reads grammar definitions -- its own language Aether, plus ABNF, BNF, EBNF, and PEG importers -- and turns each one into a working Lexer + Parser + Executor for whatever language the grammar describes, via either an interpreted VM backend (Grammar.VM) or compile-time native codegen (__using__/1 below).

The ABNF (RFC 5234 + RFC 7405) front-end: priv/grammar/abnf.aether parsed by Aether's own front-end, compiled by the native codegen backend (use Ichor), dispatching to Ichor.ABNF.Actions (Grammar.IR as its target category, same as Regex.Actions).

Turns a parsed ABNF rulelist (RFC 5234 + RFC 7405) into a real %{rule_name_atom => Grammar.IR.expr()} map -- one Grammar.IR tree per ABNF rule, mirroring how Regex.Actions turns a single /pattern/ into one Grammar.IR tree. This is an importer's own output, not a runnable Aether.Grammar -- ABNF has no lexer/rule (token/parser) split the way Aether does, so deciding which of an ABNF ruleset's productions become Aether tokens vs. rules is a separate concern this module doesn't address.

The classical BNF front-end -- no single citable standard exists; this follows the ALGOL 60 Report's own convention with quoted terminals, matching how BNF is actually written today (see Ichor.BNF.Actions's own moduledoc for the deliberate deviation that is): priv/grammar/bnf.aether parsed by Aether's own front-end, compiled by the native codegen backend (use Ichor), dispatching to Ichor.BNF.Actions (Grammar.IR as its target category).

Turns a parsed classical BNF grammar_file into a real %{nonterminal_name_atom => Grammar.IR.expr()} map -- one Grammar.IR tree per rule, the same target category Ichor.ABNF.Actions and Regex.Actions both use.

This implementation's answers to the Ichor Conformance Suite, through either backend.

An actions module answering for nothing, so every rule and token falls to Ichor.Actions's defaults: what the suite compares is the default tree.

This implementation's answers to the Ichor Conformance Suite's toolkit cases: Ichor.Toolkit.Pratt, Ichor.Backtrack.Bindings, Ichor.Backtrack.Tree and Ichor.Toolkit.TermWalk, each run on data a case gives and answering data.

The suite's terms, as Ichor.Backtrack.Term sees them: {:var, name}, {:term, functor, args}, or an atomic value.

The ISO/IEC 14977 EBNF front-end: priv/grammar/ebnf-iso.aether parsed by Aether's own front-end, compiled by the native codegen backend (use Ichor), dispatching to Ichor.EBNF.ISO.Actions (Grammar.IR as its target category).

Turns a parsed ISO/IEC 14977 EBNF syntax into a real %{meta_identifier_atom => Grammar.IR.expr()} map -- one Grammar.IR tree per rule, the same target category Ichor.ABNF.Actions and Ichor.BNF.Actions both use.

The W3C-style EBNF front-end -- the notation the XML 1.0 spec's own section 6 "Notation" uses, also shared by XQuery/XPath's grammars: priv/grammar/ebnf-w3c.aether parsed by Aether's own front-end, compiled by the native codegen backend (use Ichor), dispatching to Ichor.EBNF.W3C.Actions (Grammar.IR as its target category).

Turns a parsed W3C-style EBNF grammar_file (the notation the XML 1.0 spec's own section 6 uses, also shared by XQuery/XPath's grammars) into a real %{ident_atom => Grammar.IR.expr()} map -- one Grammar.IR tree per rule, the same target category every other importer this project builds uses.

Turns an ABNF/BNF/ISO-EBNF/W3C-EBNF/PEG ruleset (as Ichor.ABNF/ Ichor.BNF/Ichor.EBNF.ISO/Ichor.EBNF.W3C/Ichor.PEG produce them -- %{rule_name_atom => Grammar.IR.expr()}, explicitly documented as "not a runnable Aether.Grammar") into one that actually is: picks a root (the source's own first-declared rule, unless overridden), assembles a full %Aether.Grammar{}, and (for ABNF specifically) fills in RFC 5234 Appendix B's implicit "core rules" (ALPHA, DIGIT, CRLF, ...) for anything the source references but never defines itself -- extremely common in real ABNF, which generally assumes those are always available.

The PEG front-end (Ford's paper, pest/PEG.js-style convention): priv/grammar/peg.aether parsed by Aether's own front-end, compiled by the native codegen backend (use Ichor), dispatching to Ichor.PEG.Actions (Grammar.IR as its target category).

Turns a parsed PEG grammar_file (Ford's paper, pest/PEG.js-style convention) into a real %{ident_atom => Grammar.IR.expr()} map -- one Grammar.IR tree per rule, the same target category every other importer this project builds uses.

The behaviour a @refine("Module", "function", ...) token suffix dispatches to -- reclassifying (or validating/decoding) a token the Tokenizer already matched, before the Parser ever sees it. @keywords is sugar for the common table-lookup case of this same mechanism (see Grammar.Lexer); @refine is the escape hatch for anything needing real logic: escape-sequence decoding, or disambiguating a token based on what came immediately before it (JS's / being a regex-literal start or a division operator, depending on the preceding token).

Small, IR-agnostic quote/unquote mechanics -- not a code generator, and deliberately not a framework for building one (no tree-walker, no visitor dispatch, no assumed node shape, no assumed function-signature convention). Each function here solves exactly one mechanical pain point that recurs when hand-writing a compiler-to- quoted-Elixir backend, extracted from Grammar.Native's own codegen (CharCompiler, RuleCompiler, Native.LR) after the same handful of tricks turned up independently duplicated across (and within) those modules -- the same duplication signal that justified Ichor.Toolkit.Fixpoint.

The generic "repeatedly recompute a value from itself until it stops changing" primitive -- extracted from a pattern that had already been hand-written three separate times in Ichor's own compiler internals before this existed: Grammar.Analysis's private fixpoint/3 (nullable-set and always-empty-set computation, both over a MapSet.t(atom()) of rule/token names) and Grammar.LRTable.Sets' private fixpoint/2/fixpoint_map/2 (FIRST/FOLLOW-set computation, one over a MapSet, one over a Map). None of those three needed anything MapSet- or Map-specific about the iteration itself -- only about what step_fn does with the value in between -- which is exactly what this generalizes: step_fn closes over whatever fixed data it needs (a grammar's own rule list, a graph, ...), this module never sees that data at all, only the value being iterated.

Small, generic graph primitives -- extracted from Grammar.Analysis's own private reachable/2/do_reachable/3 (used to detect left recursion: a rule is left-recursive exactly when it's reachable from its own leading references). Nothing here is grammar-specific -- neighbors_fn is the caller's own adjacency, a plain function rather than a hardcoded map, so this works over whatever graph an author's own AST/IR/dependency structure represents, not just Ichor's own rule-reference graphs.

The off-side-rule (indentation-sensitive layout) algorithm Python's own tokenizer uses, and Haskell's/F#'s: given each logical line's indentation width, in order, maintain a stack of open indentation levels and emit :indent/:dedent markers as the width rises or falls -- so a parser downstream never has to know about columns at all, just consumes ordinary tokens.

A generic nested lexical scope / symbol table: a stack of name->value bindings, innermost scope first, supporting the "define here, look up through enclosing scopes" shape almost every real language's semantic analysis needs (variable resolution, nested function/block scoping, ...). No existing internal precedent to extract this from (unlike Ichor.Toolkit.Fixpoint/Graph) -- a from-scratch design, but a standard, well-understood one.

Hindley-Milner-style let-polymorphism: generalize/4 and instantiate/3, built on Ichor.Backtrack.Bindings.unify_occurs_check/4 (plain unify/4 deliberately has no occurs-check, matching ISO Prolog's own default -- a type checker needs one, or unifying a type variable with a type containing itself would silently build an infinite type instead of failing).

Runs the exact same parse -> analyze -> native-codegen pipeline as use Ichor, grammar: ..., actions: ... (Ichor.generate/3), but once, from the command line, writing the result to disk as an ordinary module instead of splicing it into a macro expansion. This is the recommended way to use Ichor for anything shipping to production -- see below for why.

Reads a .aether grammar file and lists every token it defines -- user-declared, anonymous (auto-promoted from an inline rule literal), and the five predefined tokens (always present whether overridden or left at their default) -- in token_order, the same order the Lexer's maximal-munch tie-break actually uses, alongside a rendered pattern for each (Grammar.Tokens).