Grammar.Lexer (Ichor v0.1.1)

The Lexer stage of Aether's Reader/Tokenizer/Lexer/Parser split: walks a Tokenizer's raw token stream left-to-right, applying every @keywords/@refine rule an Aether.Grammar's refiners map declares, before the Parser ever runs.

Shared by both backends (Grammar.VM calls this directly; Grammar.Native's generated tokenize/2 calls it with grammar.refiners spliced in as a literal at compile time) since this operates purely on already-produced tokens plus a small data table -- unlike the Tokenizer/Parser stages, there's no per-grammar compiled matching logic to generate here, @keywords is just a plain map and @refine just dispatches to an already-compiled module by name.

Summary

Functions

reclassify(tokens, refiners)

Reclassifies tokens in order, or reports the first @refine-rejected one. refiners empty is the overwhelmingly common case (most grammars have no @keywords/@refine at all), short-circuited without walking the list.

Functions

reclassify(tokens, refiners)

@spec reclassify([Grammar.VM.Token.t()], %{
  required(atom()) => Aether.Grammar.refiner()
}) ::
  {:ok, [Grammar.VM.Token.t()]} | {:error, Ichor.Error.t()}