Grammar.Lexer (IchorRuntime v0.1.0)

Copy Markdown View Source

The Lexer stage of Aether's Reader/Tokenizer/Lexer/Parser split: walks a Tokenizer's raw token stream left-to-right, applying every @keywords/@refine rule an Aether.Grammar's refiners map declares, before the Parser ever runs.

Shared by both backends (Grammar.VM calls this directly; Grammar.Native's generated tokenize/2 calls it with grammar.refiners spliced in as a literal at compile time) since this operates purely on already-produced tokens plus a small data table -- unlike the Tokenizer/Parser stages, there's no per-grammar compiled matching logic to generate here, @keywords is just a plain map and @refine just dispatches to an already-compiled module by name.

Summary

Types

refiner()

Mirrors Aether.Grammar.refiner/0 (dev-only, in ichor proper).

Functions

reclassify(tokens, refiners)

Reclassifies tokens in order, or reports the first @refine-rejected one. refiners empty is the overwhelmingly common case (most grammars have no @keywords/@refine at all), short-circuited without walking the list.

Types

refiner()

@type refiner() ::
  {:keywords, %{required(String.t()) => atom()}}
  | {:custom, module :: module(), function :: atom(),
     possible_names :: [atom()]}

Mirrors Aether.Grammar.refiner/0 (dev-only, in ichor proper).

Functions

reclassify(tokens, refiners)

@spec reclassify([Grammar.VM.Token.t()], %{required(atom()) => refiner()}) ::
  {:ok, [Grammar.VM.Token.t()]} | {:error, Ichor.Error.t()}

Reclassifies tokens in order, or reports the first @refine-rejected one. refiners empty is the overwhelmingly common case (most grammars have no @keywords/@refine at all), short-circuited without walking the list.