Grammar.VM.Tokenizer (Ichor v0.2.1)

Copy Markdown View Source

Tokenizes a full input string against a Grammar.VM.CharCompiler program, using maximal munch: at each position, every declared token is tried, the longest match wins, ties are broken by declaration order. Trivia/skip tokens are emitted like any other -- filtering them out is the parser's job, via the Star(skip_token) that Aether.Parser already spliced into rule bodies, not the lexer's.

custom_lexemes (from Grammar.VM.CharCompiler.compile/1) names the tokens whose entire body is a Grammar.IR.CustomLexeme -- these are dispatched straight to Ichor.CustomLexeme.scan/3 instead of run as bytecode, and can hand back a capture override (embedded in the resulting Grammar.VM.Token.capture) alongside their matched text. rule_program/context only exist to support that dispatch (building the re-lex-and-match-a-rule rule_matchers its callback receives) -- every other candidate ignores them completely.

Summary

Functions

tokenize(program, custom_lexemes, token_order, rule_program, context, input, file \\ nil)

Tokenizes input completely, or reports the first position nothing matches.

tokenize_prefix(program, custom_lexemes, token_order, rule_program, context, input)

Like tokenize/7, but tokenizes only as much of a prefix of input as it can -- stopping (successfully) the moment nothing matches, instead of failing. For Ichor.CustomLexeme's re-lex-and-match primitive: the trailing bytes after wherever a dependency rule actually stops (an interpolated string's closing }, say) generally aren't valid tokens in their own right, and were never supposed to be -- they belong to whatever's scanning the outer token, not to this inner match.

Functions

tokenize(program, custom_lexemes, token_order, rule_program, context, input, file \\ nil)

@spec tokenize(
  Grammar.VM.Program.t(),
  %{required(atom()) => Grammar.VM.CharCompiler.custom_lexeme()},
  [atom()],
  Grammar.VM.Program.t(),
  term(),
  String.t(),
  String.t() | nil
) :: {:ok, [Grammar.VM.Token.t()]} | {:error, Ichor.Error.t()}

Tokenizes input completely, or reports the first position nothing matches.

tokenize_prefix(program, custom_lexemes, token_order, rule_program, context, input)

@spec tokenize_prefix(
  Grammar.VM.Program.t(),
  %{required(atom()) => Grammar.VM.CharCompiler.custom_lexeme()},
  [atom()],
  Grammar.VM.Program.t(),
  term(),
  String.t()
) :: {:ok, [Grammar.VM.Token.t()]}

Like tokenize/7, but tokenizes only as much of a prefix of input as it can -- stopping (successfully) the moment nothing matches, instead of failing. For Ichor.CustomLexeme's re-lex-and-match primitive: the trailing bytes after wherever a dependency rule actually stops (an interpolated string's closing }, say) generally aren't valid tokens in their own right, and were never supposed to be -- they belong to whatever's scanning the outer token, not to this inner match.