Makeup v0.5.0 Makeup.Lexer.Utils View Source

Utilities to deal with lexers

Link to this section Summary

Functions

Merge adjacent tokens of the same type and with the same attributes

Turn a list of tokens into raw text. For any lexer, this should hold true

Link to this section Functions

Merge adjacent tokens of the same type and with the same attributes.

Doing this will require iterating over the list of tokens again, so only do this if you have a good reason.

Turn a list of tokens into raw text. For any lexer, this should hold true:

text |> lexer.lex |> Makeup.Lexer.Utils.unlex() == text