Char-level helpers shared by every Grammar.Native.CharCompiler-generated
function, plus the maximal-munch driver Grammar.Native.generate/2's own
lex_candidates/2 feeds into -- the Tokenizer stage of Aether's
Reader/Tokenizer/Lexer/Parser split. No captures, no ref_stack --
tokens can never contain a Capture or Indent, so a token body is
just text in, {matched text, rest} out.
Summary
Types
A character class prepared by char_class/1: its ASCII members as a
128-bit bitmap, and the rest as sorted, disjoint {lo, hi} ranges in
a tuple.
Functions
ranges, in any order, overlapping or not, prepared for in_class?/2:
an ASCII character is one bit test, and any other a binary search.
Ordered choice: tries each of funs against input in order, short-circuiting on the first success (never evaluating later alternatives, same as PEG's own ordered choice).
Whether cp is in class, a class prepared by char_class/1.
Whether cp lies in any of ranges, scanning them in turn.
Picks the longest-matching candidate at the current lexer position,
ties broken by declaration order (candidates tried in that order,
replaced only on a strictly longer match). A zero-width match is
never accepted as "the next token" -- accepting one would advance
nothing and loop forever (mirrors Grammar.VM.Tokenizer's own private
consider helper's same guard exactly); such a token can still exist
and be referenced from inside other tokens, it just can't win maximal
munch on its own.
Tokenizes input completely via maximal munch, or reports the first position nothing matches -- get_candidates rebuilds the per-token match attempts against whatever's left of the input at each position.
Like tokenize/2, but tokenizes only as much of a prefix of input
as it can -- stopping (successfully) the moment nothing matches,
instead of failing. For Ichor.CustomLexeme's re-lex-and-match
primitive: the trailing bytes after wherever a dependency rule
actually stops (an interpolated string's closing }, say) generally
aren't valid tokens in their own right, and were never supposed to be
-- they belong to whatever's scanning the outer token, not to this
inner match.
Types
@type char_fun() :: (String.t() -> char_result())
@type class() :: {non_neg_integer(), tuple()}
A character class prepared by char_class/1: its ASCII members as a
128-bit bitmap, and the rest as sorted, disjoint {lo, hi} ranges in
a tuple.
Functions
@spec advance_line_col(String.t(), pos_integer(), pos_integer()) :: {pos_integer(), pos_integer()}
@spec and_pred_char(char_fun(), String.t()) :: char_result()
@spec call_n_times_char(char_fun(), non_neg_integer(), String.t()) :: char_result()
@spec char_class([{non_neg_integer(), non_neg_integer()}]) :: class()
ranges, in any order, overlapping or not, prepared for in_class?/2:
an ASCII character is one bit test, and any other a binary search.
iex> class = Grammar.Native.Runtime.Tokenizer.char_class([{?a, ?z}, {0x4E00, 0x9FFF}])
iex> {Grammar.Native.Runtime.Tokenizer.in_class?(?q, class),
...> Grammar.Native.Runtime.Tokenizer.in_class?(0x4E2D, class),
...> Grammar.Native.Runtime.Tokenizer.in_class?(?Q, class)}
{true, true, false}@spec first_char_match([char_fun()], String.t()) :: char_result()
Ordered choice: tries each of funs against input in order, short-circuiting on the first success (never evaluating later alternatives, same as PEG's own ordered choice).
@spec in_class?(non_neg_integer(), class()) :: boolean()
Whether cp is in class, a class prepared by char_class/1.
@spec in_ranges?(non_neg_integer(), [{non_neg_integer(), non_neg_integer()}]) :: boolean()
Whether cp lies in any of ranges, scanning them in turn.
For a parser generated before char_class/1 existed. A class with
hundreds of ranges, such as a Unicode identifier class, makes every
character outside it scan them all: a generated parser now prepares
the class once with char_class/1 and asks in_class?/2.
@spec not_pred_char(char_fun(), String.t()) :: char_result()
@spec opt_char(char_fun(), String.t()) :: char_result()
@spec pick_longest([{atom(), (-> {:ok, String.t(), String.t(), term()} | :fail)}]) :: {:ok, atom(), String.t(), String.t(), term()} | :fail
Picks the longest-matching candidate at the current lexer position,
ties broken by declaration order (candidates tried in that order,
replaced only on a strictly longer match). A zero-width match is
never accepted as "the next token" -- accepting one would advance
nothing and loop forever (mirrors Grammar.VM.Tokenizer's own private
consider helper's same guard exactly); such a token can still exist
and be referenced from inside other tokens, it just can't win maximal
munch on its own.
capture is nil for an ordinary candidate, or a Grammar.IR.Custom Lexeme-matched one's explicit override -- see Grammar.VM.Token.
@spec plus_char(char_fun(), String.t()) :: char_result()
@spec rep_char( char_fun(), non_neg_integer(), non_neg_integer() | :infinity, String.t() ) :: char_result()
@spec star_char(char_fun(), String.t(), String.t()) :: char_result()
@spec tokenize( (String.t() -> [{atom(), (-> {:ok, String.t(), String.t(), term()} | :fail)}]), String.t() ) :: {:ok, [Grammar.VM.Token.t()]} | {:error, Ichor.Error.t()}
Tokenizes input completely via maximal munch, or reports the first position nothing matches -- get_candidates rebuilds the per-token match attempts against whatever's left of the input at each position.
@spec tokenize_prefix( (String.t() -> [{atom(), (-> {:ok, String.t(), String.t(), term()} | :fail)}]), String.t() ) :: {:ok, [Grammar.VM.Token.t()]}
Like tokenize/2, but tokenizes only as much of a prefix of input
as it can -- stopping (successfully) the moment nothing matches,
instead of failing. For Ichor.CustomLexeme's re-lex-and-match
primitive: the trailing bytes after wherever a dependency rule
actually stops (an interpolated string's closing }, say) generally
aren't valid tokens in their own right, and were never supposed to be
-- they belong to whatever's scanning the outer token, not to this
inner match.