Makeup v0.8.0 Makeup.Lexer behaviour View Source
A lexer turns raw source code into a list of tokens.
Link to this section Summary
Functions
Merge adjacent tokens of the same type and with the same attributes
Splits a list of tokens on newline characters ()
Merges the token values into the original string
Callbacks
Lexes a string into a list of tokens
Matches groups in a list of tokens
Postprocesses a list of tokens before matching the contained groups
Parses the given string into a parsec
result that inludes a list of tokens
Parses the smallest number of tokens that make sense.
It’s a parsec
Link to this section Functions
merge([Makeup.Lexer.Types.token()]) :: [Makeup.Lexer.Types.token()]
Merge adjacent tokens of the same type and with the same attributes.
Doing this will require iterating over the list of tokens again, so only do this if you have a good reason.
split_into_lines([Makeup.Lexer.Types.token()]) :: [[Makeup.Lexer.Types.token()]]
Splits a list of tokens on newline characters ().
The result is a list of lists of tokens with no newlines.
Merges the token values into the original string.
Inverts the ouput of a lexer. That is, if lexer
is a lexer, then:
string |> lexer.lex() |> Makeup.Lexer.unlex() == string
This only works for a correctly implemented lexer, of course. The above identity can be trated as a lexer invariant for newly implemented lexers.
Link to this section Callbacks
lex(String.t(), list()) :: [Makeup.Lexer.Types.token()]
Lexes a string into a list of tokens
match_groups([Makeup.Lexer.Types.token()], String.t()) :: [ Makeup.Lexer.Types.token() ]
Matches groups in a list of tokens.
postprocess([Makeup.Lexer.Types.token()], list()) :: [ Makeup.Lexer.Types.token() ]
Postprocesses a list of tokens before matching the contained groups.
Parses the given string into a parsec
result that inludes a list of tokens.
root_element(String.t()) :: Makeup.Lexer.Types.parsec_result()
Parses the smallest number of tokens that make sense.
It’s a parsec
.