Grammar.Native.Runtime.Tokenizer (IchorRuntime v0.1.0)

Copy Markdown View Source

Char-level helpers shared by every Grammar.Native.CharCompiler-generated function, plus the maximal-munch driver Grammar.Native.generate/2's own lex_candidates/2 feeds into -- the Tokenizer stage of Aether's Reader/Tokenizer/Lexer/Parser split. No captures, no ref_stack -- tokens can never contain a Capture or Indent, so a token body is just text in, {matched text, rest} out.

Summary

Types

char_fun()
char_result()

Functions

advance_line_col(text, line, col)
and_pred_char(fun, input)
call_n_times_char(fun, n, input)
first_char_match(list, input)

Ordered choice: tries each of funs against input in order, short-circuiting on the first success (never evaluating later alternatives, same as PEG's own ordered choice).

in_ranges?(cp, ranges)
not_pred_char(fun, input)
opt_char(fun, input)
pick_longest(candidates)

Picks the longest-matching candidate at the current lexer position, ties broken by declaration order (candidates tried in that order, replaced only on a strictly longer match). A zero-width match is never accepted as "the next token" -- accepting one would advance nothing and loop forever (mirrors Grammar.VM.Tokenizer's own private consider helper's same guard exactly); such a token can still exist and be referenced from inside other tokens, it just can't win maximal munch on its own.

plus_char(fun, input)
rep_char(fun, min, max, input)
star_char(fun, input, acc \\ "")
tokenize(get_candidates, input)

Tokenizes input completely via maximal munch, or reports the first position nothing matches -- get_candidates rebuilds the per-token match attempts against whatever's left of the input at each position.

tokenize_prefix(get_candidates, input)

Like tokenize/2, but tokenizes only as much of a prefix of input as it can -- stopping (successfully) the moment nothing matches, instead of failing. For Ichor.CustomLexeme's re-lex-and-match primitive: the trailing bytes after wherever a dependency rule actually stops (an interpolated string's closing }, say) generally aren't valid tokens in their own right, and were never supposed to be -- they belong to whatever's scanning the outer token, not to this inner match.

Types

char_fun()

@type char_fun() :: (String.t() -> char_result())

char_result()

@type char_result() :: {:ok, String.t(), String.t()} | :fail

Functions

advance_line_col(text, line, col)

@spec advance_line_col(String.t(), pos_integer(), pos_integer()) ::
  {pos_integer(), pos_integer()}

and_pred_char(fun, input)

@spec and_pred_char(char_fun(), String.t()) :: char_result()

call_n_times_char(fun, n, input)

@spec call_n_times_char(char_fun(), non_neg_integer(), String.t()) :: char_result()

first_char_match(list, input)

@spec first_char_match([char_fun()], String.t()) :: char_result()

Ordered choice: tries each of funs against input in order, short-circuiting on the first success (never evaluating later alternatives, same as PEG's own ordered choice).

in_ranges?(cp, ranges)

@spec in_ranges?(non_neg_integer(), [{non_neg_integer(), non_neg_integer()}]) ::
  boolean()

not_pred_char(fun, input)

@spec not_pred_char(char_fun(), String.t()) :: char_result()

opt_char(fun, input)

@spec opt_char(char_fun(), String.t()) :: char_result()

pick_longest(candidates)

@spec pick_longest([{atom(), (-> {:ok, String.t(), String.t(), term()} | :fail)}]) ::
  {:ok, atom(), String.t(), String.t(), term()} | :fail

Picks the longest-matching candidate at the current lexer position, ties broken by declaration order (candidates tried in that order, replaced only on a strictly longer match). A zero-width match is never accepted as "the next token" -- accepting one would advance nothing and loop forever (mirrors Grammar.VM.Tokenizer's own private consider helper's same guard exactly); such a token can still exist and be referenced from inside other tokens, it just can't win maximal munch on its own.

capture is nil for an ordinary candidate, or a Grammar.IR.Custom Lexeme-matched one's explicit override -- see Grammar.VM.Token.

plus_char(fun, input)

@spec plus_char(char_fun(), String.t()) :: char_result()

rep_char(fun, min, max, input)

@spec rep_char(
  char_fun(),
  non_neg_integer(),
  non_neg_integer() | :infinity,
  String.t()
) ::
  char_result()

star_char(fun, input, acc \\ "")

@spec star_char(char_fun(), String.t(), String.t()) :: char_result()

tokenize(get_candidates, input)

@spec tokenize(
  (String.t() ->
     [{atom(), (-> {:ok, String.t(), String.t(), term()} | :fail)}]),
  String.t()
) :: {:ok, [Grammar.VM.Token.t()]} | {:error, Ichor.Error.t()}

Tokenizes input completely via maximal munch, or reports the first position nothing matches -- get_candidates rebuilds the per-token match attempts against whatever's left of the input at each position.

tokenize_prefix(get_candidates, input)

@spec tokenize_prefix(
  (String.t() ->
     [{atom(), (-> {:ok, String.t(), String.t(), term()} | :fail)}]),
  String.t()
) :: {:ok, [Grammar.VM.Token.t()]}

Like tokenize/2, but tokenizes only as much of a prefix of input as it can -- stopping (successfully) the moment nothing matches, instead of failing. For Ichor.CustomLexeme's re-lex-and-match primitive: the trailing bytes after wherever a dependency rule actually stops (an interpolated string's closing }, say) generally aren't valid tokens in their own right, and were never supposed to be -- they belong to whatever's scanning the outer token, not to this inner match.