API Reference Ichor v#0.1.1

The semantic half of Aether's front-end: turns an Aether.Reader.Grammar CST into a fully resolved Aether.Grammar, with every body already Grammar.IR.

The fully compiled output of Aether.Parser: grammar-wide settings plus every token and rule body, each already resolved to Grammar.IR.

Turns .aether source text into a flat list of Aether.Tokens.

Parses .aether source into a fully resolved Aether.Grammar, in two stages: Aether.Reader (pure syntax -- a token/rule reference split, pragmas, the expression grammar -- into a concrete syntax tree with no desugaring) followed by Aether.Eval (that CST into Grammar.IR, handling everything that needs whole-grammar context: predefined-token override/use tracking, case-insensitivity, character-class/regex desugaring, inline-literal promotion, @skip splicing, and final @root/@skip validation).

The pure-syntax half of Aether's front-end: turns .aether source (via Aether.Lexer) into a concrete syntax tree that's a faithful record of what was written -- every pragma, token, and rule definition, in file order -- with no desugaring and no cross-definition semantics applied.

The CST Aether.Reader.read/2 produces: grammar-wide pragmas plus every token/rule definition in file order, each body an Aether.Reader.cst() tree. Aether.Eval.build/1 is the only consumer.

A single lexical token produced by Aether.Lexer from .aether source text.

The analysis pass between a front-end (Aether.Parser, or one of the ABNF/BNF/EBNF/PEG importers) and either backend (Grammar.VM or native codegen)

The interpreted graph-structured-stack backend: runs the same Grammar.LRTable SLR(1) table Grammar.LR does, but accepts conflicts instead of requiring their absence -- every action in a conflict cell is taken, forking the parse across a real Grammar.GLR.GSS (node-sharing, not independent per-branch stacks: two derivations that reach the same state at the same position merge, which is what keeps this from blowing up combinatorially on a grammar with only a handful of local conflicts). The GSS-driving loop itself lives in Grammar.GLR.Runtime, shared unchanged with Grammar.Native.GLR -- this module's own job is just building the table (recompiled on every call, the same "VM interpreted" convention Grammar.VM and Grammar.LR already follow) and wrapping its plain action/goto maps into the closures Runtime.run/6 expects.

The graph-structured stack itself: nodes keyed by {state, stream_position} (deduplicated -- two derivations that reach the same state at the same position share one node, the actual efficiency win over independent parallel stacks), edges labeled with whatever that edge's own transition produced (a shifted %Grammar.VM.Token{}, or a reduced nonterminal's own already-built captures map) plus the provenance accumulated along it so far.

The GSS-driving shift-reduce/fork loop itself, shared by the interpreted Grammar.GLR (which wraps a Grammar.LRTable's plain action/goto maps into closures) and Grammar.Native.GLR (which passes compiled per-state functions instead) -- extracted so both reuse the exact same, already-proven algorithm. The graph-structured stack itself (node merging, multi-path reduce enumeration -- Grammar.GLR.GSS) is inherently a runtime, input-driven data structure that can't compile away no matter which engine calls into it; only how a caller looks up actions/goto for a state varies.

The normalized grammar AST every Ichor front-end (Aether, ABNF, BNF, EBNF, PEG) compiles down to, and that every later stage -- the analysis pass, Grammar.VM, native codegen -- consumes instead of talking to any particular front-end's own syntax.

Positive lookahead, consumes nothing: &expr.

Matches any single character: ..

Named capture for AST construction: name:expr.

Character class, including unicode ranges: [a-z0-9_].

Ordered PEG choice: a | b | c -- first match wins, always.

@native("Module", "function", dep1, dep2, ...): a rule body that delegates to hand-written Elixir code (an Ichor.CustomRule implementation) instead of ordinary Grammar.IR combinators -- the escape hatch for constructs no static PEG grammar can express (a Prolog-style mutable operator-precedence table, C's typedef-vs- expression ambiguity, and similar "what matches here depends on something read earlier" problems).

@native("Module", "function", dep1, ...) at token position: a token whose entire body is hand-written Elixir code (an Ichor.CustomLexeme implementation) instead of ordinary character-level combinators -- the escape hatch for lexical constructs no fixed maximal-munch tokenizer can express (a heredoc's dynamic terminator, a string literal with embedded interpolated expressions).

Indentation-sensitive block: @indent(expr) / @samecol(expr).

Exact string match.

Source-location metadata carried by every Grammar.IR node, so an error raised at any later stage (analysis, either backend) can still point back at the exact line/column in the original grammar file rather than just naming an IR node in the abstract.

Negative lookahead, consumes nothing: !expr.

Zero or one: expr?.

One or more: expr+.

Bounded repetition: expr{n}, expr{n,}, expr{n,m}.

Reference to another rule (or token), by name.

Ordered sequence of sub-expressions: a b c.

Zero or more: expr*.

The deterministic bottom-up backend: builds an SLR(1) table via Grammar.LRTable and requires it be conflict-free -- a real conflict is a compile-time error here (compile/1), never something forked through (that's what Grammar.GLR is for). No graph-structured-stack bookkeeping at all: an ordinary shift-reduce loop over a single stack.

The linear LR stack's shift/reduce mechanics -- shared by the interpreted Grammar.LR and Grammar.Native.LR's per-state generated code, since a stack push/pop is exactly the same operation regardless of whether the state driving it came from a Map.get or a compiled case.

Builds an SLR(1) action/goto table from an @engine lr/@engine glr grammar -- shared by both Grammar.LR (which additionally requires the result be conflict-free) and Grammar.GLR (which accepts conflicts and forks over them at runtime).

Canonical LR(0) item-set construction (closure/goto over Grammar.LRTable.Desugar's flat production list), plus SLR(1) action/goto table construction on top of it.

Builds the raw-capture map Ichor.Actions expects ({:token,...}/ {:rule,...}/{:text,...}) from one reduced production's own RHS entries -- shared by Grammar.LR (a single linear stack) and Grammar.GLR (a graph-structured one): both reduce the same kind of production, against the same kind of token stream, needing the exact same per-position classification (see Grammar.LRTable.Production's own moduledoc for what each capture kind means), so this logic has exactly one place to live rather than two copies drifting apart.

Flattens grammar.rules' PEG-shaped IR into the flat CFG production list Grammar.LRTable.Production describes, for Grammar.LRTable's LR(0)/SLR(1) construction to consume.

One flat CFG production, lhs -> rhs, produced by Grammar.LRTable.Desugar from a grammar's PEG-shaped IR (Seq/Choice/Star/Plus/Opt/Rep each desugar to one or more of these; Choice in particular maps directly onto "more than one production for the same lhs" -- the natural CFG shape ordered PEG choice already resembles, minus the ordering).

Standard textbook nullable/FIRST/FOLLOW fixpoint algorithms, over Grammar.LRTable.Desugar's flat CFG production list rather than the original PEG IR -- recomputed fresh (not reusing Grammar.Analysis's own compute_nullable/1) since the augmented nonterminal set here includes every helper nonterminal a Star/Plus/Opt/Rep/group desugared into, which Grammar.Analysis never sees.

The Lexer stage of Aether's Reader/Tokenizer/Lexer/Parser split: walks a Tokenizer's raw token stream left-to-right, applying every @keywords/@refine rule an Aether.Grammar's refiners map declares, before the Parser ever runs.

The compile-time codegen backend: turns a validated %Aether.Grammar{} into quoted Elixir function definitions -- the same two-stage Lexer -> Parser split Grammar.VM compiles to bytecode for, but here as direct function calls a use Ichor, grammar:, actions: caller splices straight into its own module (via Ichor's __using__ macro), skipping bytecode interpretation entirely.

Compiles every token in a grammar (character-level: Literal, CharClass, Any, RuleRef to another token, and the combinators Seq/Choice/Star/Plus/Opt/Rep/AndPred/NotPred -- the same shapes Grammar.VM.CharCompiler compiles) into quoted Elixir function definitions instead of bytecode.

The compile-time codegen backend for @engine glr grammars: builds the SLR(1) table via Grammar.LRTable once, at Elixir-compile-time (conflicts are expected here, never rejected -- that's Grammar.Native.LR's job), and compiles the action/goto lookup into per-state generated function clauses instead of the interpreted Grammar.GLR's Map.gets.

The compile-time codegen backend for @engine lr grammars: builds the SLR(1) table via Grammar.LRTable once, at Elixir-compile-time (inside use Ichor's macro expansion), and requires it be conflict-free -- same requirement Grammar.LR.compile/1 enforces at runtime, just checked once here instead of on every call.

Compiles every rule in a grammar into quoted Elixir function definitions run against the lexer's token stream (never raw characters -- Aether's own two-stage split), mirroring Grammar.VM.RuleCompiler's own semantics exactly (bare-reference implicit self-capture, Indent/@samecol, capture-shape rules) but producing direct function calls instead of bytecode.

Rule-level helpers shared by every Grammar.Native.RuleCompiler-generated function -- the Parser stage of Aether's Reader/Tokenizer/Lexer/Parser split. Mirrors Grammar.VM.TokenInterpreter's own semantics exactly (same backtracking discipline, same no-progress guard on Star), just expressed as plain function composition instead of a bytecode interpreter loop -- Grammar.VM.Compiler's moduledoc describes the same combinators this module implements.

Char-level helpers shared by every Grammar.Native.CharCompiler-generated function, plus the maximal-munch driver Grammar.Native.generate/2's own lex_candidates/2 feeds into -- the Tokenizer stage of Aether's Reader/Tokenizer/Lexer/Parser split. No captures, no ref_stack -- tokens can never contain a Capture or Indent, so a token body is just text in, {matched text, rest} out.

Generates the Tokenizer -> Lexer half of a compiled grammar's tokenize/2 (char-level maximal munch via Grammar.Native.CharCompiler, then @keywords/@refine reclassification via the shared, backend-agnostic Grammar.Lexer) -- extracted once out of Grammar.Native.generate/2 so Grammar.Native.LR/Grammar.Native.GLR can reuse it unchanged: every engine consumes the exact same token stream, only the parser half differs (direct PEG combinator calls, vs. compiled LR/GLR state dispatch).

The very first, Reader-adjacent stage in front of the Tokenizer: confirms input is valid UTF-8 before any char-level matcher -- built on Elixir binary pattern matching against ::utf8 codepoints -- ever touches it. Malformed input left unchecked doesn't fail cleanly: it crashes a compiled matcher with a MatchError partway through tokenizing, deep inside either backend, instead of surfacing as an ordinary Ichor.Error.

Token introspection for a compiled Aether.Grammar -- the logic behind mix ichor.tokens. Depends only on the Aether front-end's output (token_order/tokens/anon_tokens), not the analysis pass or either backend, so it works on any grammar that parses, even one the analysis pass would otherwise reject.

The interpreted runtime backend: compiles an Aether.Grammar to bytecode and runs it against real input.

Compiles every token in a grammar (character-level: Literal, CharClass, Any, and RuleRef to another token -- the only node shapes an ordinary token body can contain) into one linked Grammar.VM.Program, run by Grammar.VM.Tokenizer.

Runs a Grammar.VM.CharCompiler-produced program against raw input text -- the machine Grammar.VM.Tokenizer uses to test one token at a given starting position.

The combinator half of Grammar.VM's bytecode compilation, shared between the character-level compiler (token bodies, run by the lexer) and the token-stream-level compiler (rule bodies, run by the parser). Seq/Choice/Star/Plus/Opt/Rep/AndPred/NotPred compile identically either way -- only the leaf nodes differ (Literal/CharClass/Any/RuleRef for tokens; RuleRef/Indent/ Capture for rules -- tokens can never contain a Capture), so each caller supplies its own leaf_fun for those.

Turns [{name, ops}] -- one label-relative op list per token or rule, as produced by Grammar.VM.Compiler -- into a single linked Grammar.VM.Program.

Linked PEG bytecode: every token (or every rule) in a grammar, compiled into one shared instruction tuple so :call can jump between them by index, plus a name -> entry-index map for looking up where to start.

Compiles every rule in a grammar into one linked Grammar.VM.Program, run by Grammar.VM.TokenInterpreter against the lexer's token stream (never against raw characters -- Aether's two-stage Lexer -> Parser split). A RuleRef compiles to {:token, name} (consume one stream token of that type) when it names a token, or {:call, name} (jump into that rule's own compiled code) when it names another rule -- the two are otherwise indistinguishable in the IR, so the grammar's own token/rule namespaces are what disambiguate.

One lexed token from Grammar.VM.Tokenizer -- the target grammar's own tokens (e.g. a calculator's NUMBER, "+"), not to be confused with Aether.Token (Ichor's front-end lexing .aether source itself).

Runs a Grammar.VM.RuleCompiler-produced program against a lexed token stream -- the machine Grammar.VM uses for the parser stage.

Tokenizes a full input string against a Grammar.VM.CharCompiler program, using maximal munch: at each position, every declared token is tried, the longest match wins, ties are broken by declaration order. Trivia/skip tokens are emitted like any other -- filtering them out is the parser's job, via the Star(skip_token) that Aether.Parser already spliced into rule bodies, not the lexer's.

Ichor reads grammar definitions -- its own language Aether, plus ABNF, BNF, EBNF, and PEG importers -- and turns each one into a working Lexer + Parser + Executor for whatever language the grammar describes, via either an interpreted VM backend (Grammar.VM) or compile-time native codegen (__using__/1 below).

The ABNF (RFC 5234 + RFC 7405) front-end: priv/grammar/abnf.aether parsed by Aether's own front-end, compiled by the native codegen backend (use Ichor), dispatching to Ichor.ABNF.Actions (Grammar.IR as its target category, same as Regex.Actions).

Turns a parsed ABNF rulelist (RFC 5234 + RFC 7405) into a real %{rule_name_atom => Grammar.IR.expr()} map -- one Grammar.IR tree per ABNF rule, mirroring how Regex.Actions turns a single /pattern/ into one Grammar.IR tree. This is an importer's own output, not a runnable Aether.Grammar -- ABNF has no lexer/rule (token/parser) split the way Aether does, so deciding which of an ABNF ruleset's productions become Aether tokens vs. rules is a separate concern this module doesn't address.

How a parsed Aether AST turns into an actual result -- a sandboxed program's final value, a config's map/struct, a query's result set, or transpiled output. One mechanism, used differently per grammar: a grammar's Actions module only implements the rules/tokens it needs custom behavior for (@optional_callbacks); everything else falls back to the defaults below.

The classical BNF front-end -- no single citable standard exists; this follows the ALGOL 60 Report's own convention with quoted terminals, matching how BNF is actually written today (see Ichor.BNF.Actions's own moduledoc for the deliberate deviation that is): priv/grammar/bnf.aether parsed by Aether's own front-end, compiled by the native codegen backend (use Ichor), dispatching to Ichor.BNF.Actions (Grammar.IR as its target category).

Turns a parsed classical BNF grammar_file into a real %{nonterminal_name_atom => Grammar.IR.expr()} map -- one Grammar.IR tree per rule, the same target category Ichor.ABNF.Actions and Regex.Actions both use.

The behaviour a lazy backtracking search engine implements -- Ichor.Backtrack.Tree (Stream-backed, correctness-first) is the only one today; a future WAM-grade engine would implement the same contract, letting whatever's built on top (a Prolog-style SLD-resolution loop, say) swap engines without changing.

A substitution: which logic variables are bound to which terms so far, opaque outside this module. unify/4 is the whole unification algorithm -- standard, recursive, structural unification with no occurs-check (matches ISO Prolog's own default: X = f(X) succeeds, building a cyclic term, rather than failing or looping to detect the cycle -- occurs-check is opt-in in real Prolog too, via unify_with_occurs_check/2, never the default).

The behaviour a caller's own term representation implements so Ichor.Backtrack.Bindings can unify it -- passed explicitly to every Bindings function (resolve/3, unify/4), the same "module passed as a plain argument, never looked up dynamically" convention Ichor.Actions.evaluate/5 already uses for an actions module.

The correctness-first Ichor.Backtrack engine: a solution is a plain thunk, (-> :empty | {:solution, value, rest}) -- a manually-unfolded lazy list (a "search tree," hence the name), not backed by any process/agent or Elixir Stream machinery. Each combinator forces exactly one step of whatever it's combining, which is what makes once/1 genuinely stop exploring rather than compute everything and discard all but the first result, and what makes an infinite Prolog recursion still yield its first solutions instead of hanging forever building a list.

Every capture an action receives bundles the raw parsed structure with a callable to evaluate it: node is always available, unevaluated -- what quote returns directly. eval is what an action calls when it actually wants a child's value, with an explicit context it controls. Never evaluated automatically -- that's the whole point: a special form like if or quote decides whether and when to call eval at all, which is exactly what lets an untaken if branch, or quote's own argument, go unevaluated (and unbound-symbol/nonterminating errors inside them stay latent) instead of always eagerly running.

The behaviour a Grammar.IR.CustomLexeme @native("Module", "function", ...) node (at token position) dispatches to -- the escape hatch for lexical constructs a fixed maximal-munch tokenizer can't express: a heredoc's dynamic terminator, a string literal with embedded interpolated expressions, a \catcode-style mid-scan reconfiguration.

The behaviour a Grammar.IR.Custom @native("Module", "function", ...) node dispatches to -- the escape hatch for grammar constructs no static PEG grammar can express: a Prolog-style mutable operator-precedence table, C's typedef-vs-expression ambiguity, and similar "what matches here depends on something read earlier" problems.

The ISO/IEC 14977 EBNF front-end: priv/grammar/ebnf-iso.aether parsed by Aether's own front-end, compiled by the native codegen backend (use Ichor), dispatching to Ichor.EBNF.ISO.Actions (Grammar.IR as its target category).

Turns a parsed ISO/IEC 14977 EBNF syntax into a real %{meta_identifier_atom => Grammar.IR.expr()} map -- one Grammar.IR tree per rule, the same target category Ichor.ABNF.Actions and Ichor.BNF.Actions both use.

The W3C-style EBNF front-end -- the notation the XML 1.0 spec's own section 6 "Notation" uses, also shared by XQuery/XPath's grammars: priv/grammar/ebnf-w3c.aether parsed by Aether's own front-end, compiled by the native codegen backend (use Ichor), dispatching to Ichor.EBNF.W3C.Actions (Grammar.IR as its target category).

Turns a parsed W3C-style EBNF grammar_file (the notation the XML 1.0 spec's own section 6 uses, also shared by XQuery/XPath's grammars) into a real %{ident_atom => Grammar.IR.expr()} map -- one Grammar.IR tree per rule, the same target category every other importer this project builds uses.

One error struct, reused across every stage -- Lexer, Parser, analysis pass, and Ichor.Actions -- so errors look the same regardless of origin.

The default-fallback shape Ichor.Actions builds for a matched rule that has more than one meaningful capture (a rule with exactly one capture passes that capture's own value straight through instead): rule names the matched rule, captures maps each capture name to its already-evaluated value (or a list of values, for a name captured more than once -- e.g. inside a *), and span locates the whole match back in the original input.

The PEG front-end (Ford's paper, pest/PEG.js-style convention): priv/grammar/peg.aether parsed by Aether's own front-end, compiled by the native codegen backend (use Ichor), dispatching to Ichor.PEG.Actions (Grammar.IR as its target category).

Turns a parsed PEG grammar_file (Ford's paper, pest/PEG.js-style convention) into a real %{ident_atom => Grammar.IR.expr()} map -- one Grammar.IR tree per rule, the same target category every other importer this project builds uses.

The behaviour a @refine("Module", "function", ...) token suffix dispatches to -- reclassifying (or validating/decoding) a token the Tokenizer already matched, before the Parser ever sees it. @keywords is sugar for the common table-lookup case of this same mechanism (see Grammar.Lexer); @refine is the escape hatch for anything needing real logic: escape-sequence decoding, or disambiguating a token based on what came immediately before it (JS's / being a regex-literal start or a division operator, depending on the preceding token).

Small, IR-agnostic quote/unquote mechanics -- not a code generator, and deliberately not a framework for building one (no tree-walker, no visitor dispatch, no assumed node shape, no assumed function-signature convention). Each function here solves exactly one mechanical pain point that recurs when hand-writing a compiler-to- quoted-Elixir backend, extracted from Grammar.Native's own codegen (CharCompiler, RuleCompiler, Native.LR) after the same handful of tricks turned up independently duplicated across (and within) those modules -- the same duplication signal that justified Ichor.Toolkit.Fixpoint.

The generic "repeatedly recompute a value from itself until it stops changing" primitive -- extracted from a pattern that had already been hand-written three separate times in Ichor's own compiler internals before this existed: Grammar.Analysis's private fixpoint/3 (nullable-set and always-empty-set computation, both over a MapSet.t(atom()) of rule/token names) and Grammar.LRTable.Sets' private fixpoint/2/fixpoint_map/2 (FIRST/FOLLOW-set computation, one over a MapSet, one over a Map). None of those three needed anything MapSet- or Map-specific about the iteration itself -- only about what step_fn does with the value in between -- which is exactly what this generalizes: step_fn closes over whatever fixed data it needs (a grammar's own rule list, a graph, ...), this module never sees that data at all, only the value being iterated.

Small, generic graph primitives -- extracted from Grammar.Analysis's own private reachable/2/do_reachable/3 (used to detect left recursion: a rule is left-recursive exactly when it's reachable from its own leading references). Nothing here is grammar-specific -- neighbors_fn is the caller's own adjacency, a plain function rather than a hardcoded map, so this works over whatever graph an author's own AST/IR/dependency structure represents, not just Ichor's own rule-reference graphs.

The off-side-rule (indentation-sensitive layout) algorithm Python's own tokenizer uses, and Haskell's/F#'s: given each logical line's indentation width, in order, maintain a stack of open indentation levels and emit :indent/:dedent markers as the width rises or falls -- so a parser downstream never has to know about columns at all, just consumes ordinary tokens.

Precedence-climbing (Pratt) expression parsing over a runtime-mutable operator table -- extracted from a pattern already hand-rolled once, OpExprTest.Operators.climb/6 (test/support/opexpr_operators.ex), the worked example proving Grammar.IR.Custom/@native(...) rule- position dispatch works at all. That fixture only ever needed infix chains; this generalizes to prefix and postfix too, since those are exactly what Track 1's own motivating scenarios need (Prolog's op/3 supports fy/fx prefix and xf/yf postfix operators alongside infix ones, Haskell fixity declarations are infix-only but mixfix notation is not).

The generic "walk a collection, threading an accumulator through a fallible step, stopping at the first failure" primitive -- extracted from a pattern independently hand-rolled at least ten times across Ichor's own compiler internals before this existed: build_ruleset/1 (duplicated near-verbatim across Ichor.EBNF.ISO.Actions, Ichor.EBNF.W3C.Actions, Ichor.BNF.Actions, Ichor.ABNF.Actions, and Ichor.PEG.Actions), Ichor.Backtrack.Bindings' own unify_compound/5 inner reduce over zipped compound-term arguments (whose failure sentinel is a bare :fail, not {:error, _}), Aether.Eval's own process_defs/2, and -- byte-for-byte identical to each other -- Ichor.Actions' own eval_one/2 list-of-captures branch, and Aether.Eval's own convert_list/3 and convert_seq_terms/3.

A generic nested lexical scope / symbol table: a stack of name->value bindings, innermost scope first, supporting the "define here, look up through enclosing scopes" shape almost every real language's semantic analysis needs (variable resolution, nested function/block scoping, ...). No existing internal precedent to extract this from (unlike Ichor.Toolkit.Fixpoint/Graph) -- a from-scratch design, but a standard, well-understood one.

Generic recursion over any Ichor.Backtrack.Term implementation -- extracted from a pattern independently duplicated inside Ichor.Toolkit.TypeScheme itself: resolve_deep/3 and the private substitute/3 shared the identical "apply a transform, then (if the result is compound) deconstruct, recurse into every argument with the same transform, and reconstruct" recursive skeleton, differing only in what the transform itself does at each step (chase a variable's binding one level, or substitute a specific variable for a fresh one); free_vars/3 is the same recursive shape again, but folding into an accumulator instead of rebuilding a term.

Hindley-Milner-style let-polymorphism: generalize/4 and instantiate/3, built on Ichor.Backtrack.Bindings.unify_occurs_check/4 (plain unify/4 deliberately has no occurs-check, matching ISO Prolog's own default -- a type checker needs one, or unifying a type variable with a type containing itself would silently build an infinite type instead of failing).

Reads a .aether grammar file and lists every token it defines -- user-declared, anonymous (auto-promoted from an inline rule literal), and the five predefined tokens (always present whether overridden or left at their default) -- in token_order, the same order the Lexer's maximal-munch tie-break actually uses, alongside a rendered pattern for each (Grammar.Tokens).