The very first, Reader-adjacent stage in front of the Tokenizer:
confirms input is valid UTF-8 before any char-level matcher --
built on Elixir binary pattern matching against ::utf8 codepoints --
ever touches it. Malformed input left unchecked doesn't fail cleanly:
it crashes a compiled matcher with a MatchError partway through
tokenizing, deep inside either backend, instead of surfacing as an
ordinary Ichor.Error.
Shared by both backends (Grammar.VM.parse/3/run_sequence/4 call
this directly; Grammar.Native's generated tokenize/2 calls it too)
since, like Grammar.Lexer, this is pure data validation with nothing
grammar-specific to compile.
Summary
Functions
error -- or each of a list of them -- placed at the end of input
when it says nowhere, and shown: given the line of input it is on,
and a caret, when it has a position but no snippet.
Where position pos of stream is, as {line, column}: where its
token starts, or, past the last token, the end of input.
tokens without any skip token: the stream an LR or GLR parse runs
on. Their rules never mention the skip token -- @skip makes it
optional between any two elements, which a table with one token of
lookahead cannot decide -- so it is dropped before parsing instead.
nil, for @noskip, changes nothing.
tokens without the skip tokens before the first other token and
after the last.
Returns input unchanged if it's valid UTF-8, or an Ichor.Error (stage: :lexer) otherwise.
Functions
@spec placed(Ichor.Error.t() | [Ichor.Error.t()], String.t()) :: Ichor.Error.t() | [Ichor.Error.t()]
error -- or each of a list of them -- placed at the end of input
when it says nowhere, and shown: given the line of input it is on,
and a caret, when it has a position but no snippet.
For a caller that gets its error from somewhere that never saw the
input: Grammar.GLR.Runtime builds "unexpected end of input" from
the token stream alone, and its callers, which have the input, place
it here.
@spec position(tuple(), non_neg_integer(), String.t()) :: {pos_integer(), pos_integer()}
Where position pos of stream is, as {line, column}: where its
token starts, or, past the last token, the end of input.
An error always says where. "Unexpected end of input" and a root
that matched nothing are errors about a place too -- the end, or the
first token -- and a user handed one with no line and column has
nowhere to look. No input at all is {1, 1}. Columns count
codepoints, as the tokenizer does.
@spec significant([Grammar.VM.Token.t()], atom() | nil) :: [Grammar.VM.Token.t()]
tokens without any skip token: the stream an LR or GLR parse runs
on. Their rules never mention the skip token -- @skip makes it
optional between any two elements, which a table with one token of
lookahead cannot decide -- so it is dropped before parsing instead.
nil, for @noskip, changes nothing.
The text of the skip tokens dropped before a token is kept on it
(trivia_before), so a capture over several tokens still holds the
input between its first and last, skip tokens included.
@spec trimmed([Grammar.VM.Token.t()], atom() | nil) :: [Grammar.VM.Token.t()]
tokens without the skip tokens before the first other token and
after the last.
Trivia around the input is not part of it. Skip-splicing puts a
skip* in front of every sequence element but the first, so
nothing inside a grammar ever expects a skip token at the very start
of the input or after its very end. Every engine and both backends
trim the stream here before matching the root, so a document's
leading newline or trailing space means the same to all of them.
The skip tokens between two others are left alone: those are the
grammar's.
skip is the grammar's @skip token, or nil for @noskip, which
changes nothing.
@spec validate(String.t()) :: {:ok, String.t()} | {:error, Ichor.Error.t()}
Returns input unchanged if it's valid UTF-8, or an Ichor.Error (stage: :lexer) otherwise.