Compiles every token in a grammar (character-level: Literal,
CharClass, Any, RuleRef to another token, and the combinators
Seq/Choice/Star/Plus/Opt/Rep/AndPred/NotPred -- the
same shapes Grammar.VM.CharCompiler compiles) into quoted Elixir
function definitions instead of bytecode.
Every generated function has the shape (input :: binary) -> {:ok, text :: binary, rest :: binary} | :fail. Each IR node gets its own
named function (a declared token's own name for its top-level node,
a fresh counter-based name for every anonymous sub-expression) --
composed via direct function calls, never inline-threaded variables
spliced across separately-quoted fragments, which sidesteps Elixir
macro hygiene entirely for everything except the handful of
combinators (Seq's chain) that must build a call sequence whose
length isn't known until compile time; those use Macro.var(name, nil) explicitly so every fragment agrees on the same variable.
A token whose entire body is a Grammar.IR.CustomLexeme
(@native(...) at token position) is pulled out separately instead of
compiled to a function here -- Grammar.Native.generate/2 dispatches
to it directly by name (see Grammar.VM.CharCompiler's own moduledoc
for why it can't be composed inside a larger token or referenced from
one).
Summary
Functions
Compiles every ordinary declared token into a list of quoted defp definitions, one named token_fn_name/1 per token plus one per anonymous sub-expression, and separately returns every Grammar.IR.CustomLexeme-bodied token's own module/function/deps.
The generated function name for a given token name -- exposed so Grammar.Native can reference a token's own matcher when generating the lexer's maximal-munch driver.
Types
Functions
@spec compile(%{required(atom()) => Grammar.IR.expr()}) :: {[Macro.t()], %{required(atom()) => custom_lexeme()}}
Compiles every ordinary declared token into a list of quoted defp definitions, one named token_fn_name/1 per token plus one per anonymous sub-expression, and separately returns every Grammar.IR.CustomLexeme-bodied token's own module/function/deps.
The generated function name for a given token name -- exposed so Grammar.Native can reference a token's own matcher when generating the lexer's maximal-munch driver.