Ichor.TokenRefiner behaviour (Ichor v0.2.0)

Copy Markdown View Source

The behaviour a @refine("Module", "function", ...) token suffix dispatches to -- reclassifying (or validating/decoding) a token the Tokenizer already matched, before the Parser ever sees it. @keywords is sugar for the common table-lookup case of this same mechanism (see Grammar.Lexer); @refine is the escape hatch for anything needing real logic: escape-sequence decoding, or disambiguating a token based on what came immediately before it (JS's / being a regex-literal start or a division operator, depending on the preceding token).

refine/4 receives:

  • raw_name/raw_text -- the token exactly as the Tokenizer matched it.
  • pos -- {line, column}.
  • preceding -- every token already reclassified so far (final form, not raw) -- what makes the JS / case work: by the time a / is being refined, whatever came immediately before it already reflects its own final classification, not a pre-lexer guess.

Returns {:ok, new_name, value} -- value becomes this token's capture override (see Grammar.VM.Token), same mechanism Ichor.CustomLexeme uses, so a decoded escape sequence (say) reaches Ichor.Actions as the token's actual value instead of its raw source text -- or {:error, reason}, surfaced as an Ichor.Error with stage: :lexer.

Like Ichor.CustomRule/Ichor.CustomLexeme, @refine(...) names its own callback function explicitly, so it doesn't have to be called refine -- @behaviour Ichor.TokenRefiner + @impl true is a convenience for a module that happens to name it that (the common case), not a requirement.

Summary

Callbacks

refine(raw_name, raw_text, pos, preceding)

@callback refine(
  raw_name :: atom(),
  raw_text :: String.t(),
  pos :: {pos_integer(), pos_integer()},
  preceding :: [Grammar.VM.Token.t()]
) :: {:ok, new_name :: atom(), value :: term()} | {:error, reason :: String.t()}