Reads a .aether grammar file and lists every token it defines --
user-declared, anonymous (auto-promoted from an inline rule literal),
and the five predefined tokens (always present whether overridden or
left at their default) -- in token_order, the same order the Lexer's
maximal-munch tie-break actually uses, alongside a rendered pattern for
each (Grammar.Tokens).
Only needs the Aether front-end to run -- no analysis pass, no VM/native backend -- so it works on any grammar that parses, even one that would later fail the left-recursion/reference-check analysis pass.
$ mix ichor.tokens sql.aether
# NAME KIND PATTERN
1 SELECT declared "SELECT"
2 FROM declared "FROM"
...
15 SPACE declared [ \t\n]+
16 DIGIT predefined [0-9]
...