Makeup v0.5.0 Makeup.Lexer.Utils View Source
Utilities to deal with lexers
Link to this section Summary
Functions
Merge adjacent tokens of the same type and with the same attributes
Turn a list of tokens into raw text. For any lexer, this should hold true
Link to this section Functions
Merge adjacent tokens of the same type and with the same attributes.
Doing this will require iterating over the list of tokens again, so only do this if you have a good reason.
Turn a list of tokens into raw text. For any lexer, this should hold true:
text |> lexer.lex |> Makeup.Lexer.Utils.unlex() == text