Grammar.Native.CharCompiler (Ichor v0.2.0)

Copy Markdown View Source

Compiles every token in a grammar (character-level: Literal, CharClass, Any, RuleRef to another token, and the combinators Seq/Choice/Star/Plus/Opt/Rep/AndPred/NotPred -- the same shapes Grammar.VM.CharCompiler compiles) into quoted Elixir function definitions instead of bytecode.

Every generated function has the shape (input :: binary) -> {:ok, text :: binary, rest :: binary} | :fail. Each IR node gets its own named function (a declared token's own name for its top-level node, a fresh counter-based name for every anonymous sub-expression) -- composed via direct function calls, never inline-threaded variables spliced across separately-quoted fragments, which sidesteps Elixir macro hygiene entirely for everything except the handful of combinators (Seq's chain) that must build a call sequence whose length isn't known until compile time; those use Macro.var(name, nil) explicitly so every fragment agrees on the same variable.

A token whose entire body is a Grammar.IR.CustomLexeme (@native(...) at token position) is pulled out separately instead of compiled to a function here -- Grammar.Native.generate/2 dispatches to it directly by name (see Grammar.VM.CharCompiler's own moduledoc for why it can't be composed inside a larger token or referenced from one).

Summary

Functions

Compiles every ordinary declared token into a list of quoted defp definitions, one named token_fn_name/1 per token plus one per anonymous sub-expression, and separately returns every Grammar.IR.CustomLexeme-bodied token's own module/function/deps.

The generated function name for a given token name -- exposed so Grammar.Native can reference a token's own matcher when generating the lexer's maximal-munch driver.

Types

custom_lexeme()

@type custom_lexeme() :: {module :: module(), function :: atom(), deps :: [atom()]}

Functions

compile(tokens)

@spec compile(%{required(atom()) => Grammar.IR.expr()}) ::
  {[Macro.t()], %{required(atom()) => custom_lexeme()}}

Compiles every ordinary declared token into a list of quoted defp definitions, one named token_fn_name/1 per token plus one per anonymous sub-expression, and separately returns every Grammar.IR.CustomLexeme-bodied token's own module/function/deps.

fn_name(name)

@spec fn_name(atom()) :: atom()

The generated function name for a given token name -- exposed so Grammar.Native can reference a token's own matcher when generating the lexer's maximal-munch driver.