Grammar.Tokens (Ichor v0.2.0)

Copy Markdown View Source

Token introspection for a compiled Aether.Grammar -- the logic behind mix ichor.tokens. Depends only on the Aether front-end's output (token_order/tokens/anon_tokens), not the analysis pass or either backend, so it works on any grammar that parses, even one the analysis pass would otherwise reject.

Summary

Types

entry()
kind()

Functions

describe(expr)

Renders a Grammar.IR expression as a compact PEG-like pattern string, for display only -- not a serialization format, and not fed back into the parser anywhere.

list(grammar)

Every token in grammar, in token_order (the same order the Lexer's maximal-munch tie-break uses) -- user-declared tokens, tokens auto-promoted from an inline rule literal (kind: :anonymous), and the five always-present predefined tokens (kind: :predefined), whether overridden or left at their default.

Types

entry()

@type entry() :: %{name: atom(), kind: kind(), pattern: String.t()}

kind()

@type kind() :: :declared | :anonymous | :predefined

Functions

describe(expr)

@spec describe(Grammar.IR.expr()) :: String.t()

Renders a Grammar.IR expression as a compact PEG-like pattern string, for display only -- not a serialization format, and not fed back into the parser anywhere.

iex> Grammar.Tokens.describe(Grammar.IR.literal("SELECT"))
"\"SELECT\""

iex> Grammar.Tokens.describe(Grammar.IR.char_class([{?0, ?9}, {?a, ?f}]))
"[0-9a-f]"

iex> digit = Grammar.IR.char_class([{?0, ?9}])
iex> Grammar.Tokens.describe(Grammar.IR.seq([Grammar.IR.plus(digit), Grammar.IR.opt(Grammar.IR.literal("."))]))
"[0-9]+ \".\"?"

iex> Grammar.Tokens.describe(Grammar.IR.choice([Grammar.IR.literal("+"), Grammar.IR.literal("-")]))
"\"+\" | \"-\""

list(grammar)

@spec list(Aether.Grammar.t()) :: [entry()]

Every token in grammar, in token_order (the same order the Lexer's maximal-munch tie-break uses) -- user-declared tokens, tokens auto-promoted from an inline rule literal (kind: :anonymous), and the five always-present predefined tokens (kind: :predefined), whether overridden or left at their default.