Grammar.VM (Ichor v0.2.0)

The interpreted runtime backend: compiles an Aether.Grammar to bytecode and runs it against real input.

Two genuinely separate compiled programs are involved, matching Aether's own "every grammar compiles to a Lexer feeding a Parser, never a scannerless single pass" design: Grammar.VM.CharCompiler + Grammar.VM.Tokenizer turn the input string into a token stream (maximal munch over every declared token); Grammar.VM.RuleCompiler + Grammar.VM.TokenInterpreter then run the rules over that stream, never over raw characters -- building a raw capture tree as they go.

parse/3 is a bare recognizer (does grammar.root match input, full stop) with no Ichor.Actions involved. run/4 is the actions-aware entry point: it matches, then hands the raw capture tree to Ichor.Actions.evaluate/5 for a given Actions module and initial context, producing the grammar's actual result (a sandboxed program's value, a config map, ...) instead of just a yes/no. run_sequence/4 is for a source that's a sequence of top-level matches against grammar.root rather than one single match spanning the whole input (loading a standard-library file one top-level form at a time, each one threading context into the next, is the motivating case) -- most grammars only ever need run/4.

context reaches the match phase too now (parse/3's third argument, run/4's initial_context, run_sequence/4's per-form-accumulated context for rule-level matching), read-only -- the one thing that ever consults it there is a Grammar.IR.Custom @native(...) node, via Ichor.CustomRule.match/4, or a Grammar.IR.CustomLexeme one via Ichor.CustomLexeme.scan/3. Nothing else in Grammar.VM.TokenInterpreter/Grammar.VM.Tokenizer reads it; only Ichor.Actions.evaluate/5 ever produces a new one.

One real gap for CustomLexeme specifically: run_sequence/4 tokenizes the entire input once, up front, with only initial_context -- unlike rule-level matching (re-run per top-level form, seeing that form's own accumulated context), a token that reads context can't see anything a later form's own evaluation produced. Fine for heredocs/string-interpolation (context there is either unused or fixed for the whole file); a hazard only for something \catcode-like that needs re-tokenization as context evolves mid-sequence -- not something run_sequence/4 supports today.

Summary

Functions

parse(grammar, input, context \\ nil)

Compiles grammar and matches it against input, requiring the root rule to consume the entire (tokenized) input. Returns the number of tokens consumed on success -- a bare recognizer result, no Ichor.Actions involved.

run(grammar, input, actions_module, initial_context \\ nil)

Like parse/2, but runs the match through actions_module (a Ichor.Actions implementation) starting from initial_context, returning the grammar's actual evaluated result.

run_sequence(grammar, input, actions_module, initial_context)

Evaluates input as a sequence of top-level matches against grammar.root, one after another (skipping only the grammar's own @skip trivia between them, the same as it would between any two ordinary tokens), threading context from each into the next. Returns every top-level value, in order, plus the final context.

Functions

parse(grammar, input, context \\ nil)

@spec parse(Aether.Grammar.t(), String.t(), term()) ::
  {:ok, non_neg_integer()} | {:error, Ichor.Error.t()}

run(grammar, input, actions_module, initial_context \\ nil)

@spec run(Aether.Grammar.t(), String.t(), module(), Ichor.Actions.context()) ::
  {:ok, term()} | {:error, Ichor.Error.t() | [Ichor.Error.t()]}

Like parse/2, but runs the match through actions_module (a Ichor.Actions implementation) starting from initial_context, returning the grammar's actual evaluated result.

run_sequence(grammar, input, actions_module, initial_context)

@spec run_sequence(Aether.Grammar.t(), String.t(), module(), Ichor.Actions.context()) ::
  {:ok, [term()], Ichor.Actions.context()}
  | {:error, Ichor.Error.t() | [Ichor.Error.t()]}