Makeup v1.0.1 Makeup.Lexer behaviour View Source

A lexer turns raw source code into a list of tokens.

Link to this section Summary

Functions

Merge adjacent tokens of the same type and with the same attributes.

Splits a list of tokens on newline characters ().

Merges the token values into the original string.

Callbacks

Lexes a string into a list of tokens

Matches groups in a list of tokens.

Postprocesses a list of tokens before matching the contained groups.

Parses the given string into a parsec result that inludes a list of tokens.

Parses the smallest number of tokens that make sense. It's a parsec.

Link to this section Functions

Merge adjacent tokens of the same type and with the same attributes.

Doing this will require iterating over the list of tokens again, so only do this if you have a good reason.

Link to this function

split_into_lines(tokens)

View Source
split_into_lines([Makeup.Lexer.Types.token()]) :: [[Makeup.Lexer.Types.token()]]

Splits a list of tokens on newline characters ().

The result is a list of lists of tokens with no newlines.

Merges the token values into the original string.

Inverts the ouput of a lexer. That is, if lexer is a lexer, then:

string |> lexer.lex() |> Makeup.Lexer.unlex() == string

This only works for a correctly implemented lexer, of course. The above identity can be trated as a lexer invariant for newly implemented lexers.

Link to this section Callbacks

Lexes a string into a list of tokens

Matches groups in a list of tokens.

Postprocesses a list of tokens before matching the contained groups.

Parses the given string into a parsec result that inludes a list of tokens.

Parses the smallest number of tokens that make sense. It's a parsec.