Grammar.Source (IchorRuntime v0.1.0)

Copy Markdown View Source

The very first, Reader-adjacent stage in front of the Tokenizer: confirms input is valid UTF-8 before any char-level matcher -- built on Elixir binary pattern matching against ::utf8 codepoints -- ever touches it. Malformed input left unchecked doesn't fail cleanly: it crashes a compiled matcher with a MatchError partway through tokenizing, deep inside either backend, instead of surfacing as an ordinary Ichor.Error.

Shared by both backends (Grammar.VM.parse/3/run_sequence/4 call this directly; Grammar.Native's generated tokenize/2 calls it too) since, like Grammar.Lexer, this is pure data validation with nothing grammar-specific to compile.

Summary

Functions

Returns input unchanged if it's valid UTF-8, or an Ichor.Error (stage: :lexer) otherwise.

Functions

validate(input)

@spec validate(String.t()) :: {:ok, String.t()} | {:error, Ichor.Error.t()}

Returns input unchanged if it's valid UTF-8, or an Ichor.Error (stage: :lexer) otherwise.