Makeup v0.8.0 Makeup.Lexer behaviour View Source

A lexer turns raw source code into a list of tokens.

Link to this section Summary

Functions

Merge adjacent tokens of the same type and with the same attributes

Splits a list of tokens on newline characters ()

Merges the token values into the original string

Callbacks

Lexes a string into a list of tokens

Matches groups in a list of tokens

Postprocesses a list of tokens before matching the contained groups

Parses the given string into a parsec result that inludes a list of tokens

Parses the smallest number of tokens that make sense. It’s a parsec

Link to this section Functions

Merge adjacent tokens of the same type and with the same attributes.

Doing this will require iterating over the list of tokens again, so only do this if you have a good reason.

Link to this function split_into_lines(tokens) View Source
split_into_lines([Makeup.Lexer.Types.token()]) :: [[Makeup.Lexer.Types.token()]]

Splits a list of tokens on newline characters ().

The result is a list of lists of tokens with no newlines.

Merges the token values into the original string.

Inverts the ouput of a lexer. That is, if lexer is a lexer, then:

string |> lexer.lex() |> Makeup.Lexer.unlex() == string

This only works for a correctly implemented lexer, of course. The above identity can be trated as a lexer invariant for newly implemented lexers.

Link to this section Callbacks

Lexes a string into a list of tokens

Matches groups in a list of tokens.

Postprocesses a list of tokens before matching the contained groups.

Parses the given string into a parsec result that inludes a list of tokens.

Parses the smallest number of tokens that make sense. It’s a parsec.