Grammar.Native.TokenizerCompiler (Ichor v0.2.0)

Copy Markdown View Source

Generates the Tokenizer -> Lexer half of a compiled grammar's tokenize/2 (char-level maximal munch via Grammar.Native.CharCompiler, then @keywords/@refine reclassification via the shared, backend-agnostic Grammar.Lexer) -- extracted once out of Grammar.Native.generate/2 so Grammar.Native.LR/Grammar.Native.GLR can reuse it unchanged: every engine consumes the exact same token stream, only the parser half differs (direct PEG combinator calls, vs. compiled LR/GLR state dispatch).

The caller supplies its own alias Grammar.Native.Runtime.Tokenizer (and Grammar.VM.Token, for the @spec) -- this module only builds the quoted defs, it doesn't own the generated module's own alias list.

Summary

Functions

generate(grammar)

Returns {support_defs, tokenize_def}: support_defs are private helpers (lex_candidates/2 plus every compiled char-matcher function) to splice in alongside the caller's own parser defs; tokenize_def is the public def tokenize/2 itself.

has_customlexeme_deps?(grammar)

Does grammar have any token-position @native(...) (Grammar.IR.CustomLexeme) with a non-empty dependency list? Those dependencies need a "re-lex and match a rule" primitive that only exists against Grammar.Native.RuleCompiler's own per-rule generated functions (ordinary PEG codegen, one function per rule) -- Grammar.Native.LR/.GLR compile rules into per-automaton-state functions instead, which have no such per-rule entry point. When this is true, the caller (Grammar.Native.LR/.GLR) splices in Grammar.Native.RuleCompiler.compile/1's ordinary PEG rule functions too -- unused by its own parse/run, present only so a dependency reference resolves -- exactly mirroring how the interpreted Grammar.LR/Grammar.GLR already compile a full Grammar.VM.RuleCompiler program purely for this same purpose, even though their own top-level parse never runs it.

Functions

generate(grammar)

@spec generate(Aether.Grammar.t()) :: {[Macro.t()], Macro.t()}

Returns {support_defs, tokenize_def}: support_defs are private helpers (lex_candidates/2 plus every compiled char-matcher function) to splice in alongside the caller's own parser defs; tokenize_def is the public def tokenize/2 itself.

has_customlexeme_deps?(grammar)

@spec has_customlexeme_deps?(Aether.Grammar.t()) :: boolean()

Does grammar have any token-position @native(...) (Grammar.IR.CustomLexeme) with a non-empty dependency list? Those dependencies need a "re-lex and match a rule" primitive that only exists against Grammar.Native.RuleCompiler's own per-rule generated functions (ordinary PEG codegen, one function per rule) -- Grammar.Native.LR/.GLR compile rules into per-automaton-state functions instead, which have no such per-rule entry point. When this is true, the caller (Grammar.Native.LR/.GLR) splices in Grammar.Native.RuleCompiler.compile/1's ordinary PEG rule functions too -- unused by its own parse/run, present only so a dependency reference resolves -- exactly mirroring how the interpreted Grammar.LR/Grammar.GLR already compile a full Grammar.VM.RuleCompiler program purely for this same purpose, even though their own top-level parse never runs it.