View Source Minipeg (Minipeg v0.3.1)
Minipeg is a minimal Parse Expression Grammars (PEG) Library
Here is a first taste of how to use it:
iex(1)> an_a_parser = char_parser("a")
...(1)> parse_string(an_a_parser, "a")
{:ok, "a"}
iex(2)> an_a_parser = char_parser("a")
...(2)> parse_string(an_a_parser, "b")
{:error, "b not member of \"a\" in char_parser(\"a\")"}
The first thing to note here is that in these doctests we have imported functions from Minipeg as follows, see the moduledocs of the corresponding modules for details
import Minipeg.Parser, only: [parse_string: 2]
import Minipeg.{Combinators, Parsers}
Basic Usage
Quite a small subset of predefined parsers and combinators would suffice to parse any context free language, however many of the patterns used to parse programming languages, little languages or small languages that exceed the practicallity of regular expressions are quite verbose.
Therefore Minipeg predefines parsers that can be easily parametrized. These parsers will be described in the utility parsers section.
Parsing Single Characters
With a character we do indeed mean a UTF-8 Code Point
The most basic parser is the...
char_parser
... which parses any character
iex(3)> parse_string(char_parser(), "h")
{:ok, "h"}
iex(4)> parse_string(char_parser(), "é")
{:ok, "é"}
iex(5)> parse_string(char_parser(), "✓")
{:ok, "✓"}
iex(6)> parse_string(char_parser(), "")
{:error, "encountered end of input in char_parser()"}
It can however be parametrized to parse only characters of a given set, this set can be provided as
a String
or an Enumerable
iex(7)> parse_string(char_parser("ab"), "b")
{:ok, "b"}
iex(8)> parse_string(char_parser(["b", "c"]), "b")
{:ok, "b"}
iex(9)> parse_string(char_parser(["b", "c"]), "a")
{:error, "a not member of \"bc\" in char_parser([\"b\", \"c\"])"}
Often used charsets might be extremly large to be defined and therefore some more specialised parsers have been defined:
Parsing POSIX character classes
If instead of a string or list we pass an atom into char_parser
it only parses a character if it matches a character class as defined in POSIX regular expressions,
which are also described in the docs of the Regex
module, here are the currently supported values:
:alnum | :alpha | :blank | :cntrl | :digit | :graph | :lower | :print | :punct | :space | :upper | :word | :xdigit
iex(10)> parser = char_parser(:alnum)
...(10)> "aD7_%"
...(10)> |> String.graphemes
...(10)> |> Enum.map(&parse_string(parser, &1))
[
ok: "a",
ok: "D",
ok: "7",
error: "Regex ~r/[[:alnum:]]/ does not match \"_\" in char_parser(:alnum)",
error: "Regex ~r/[[:alnum:]]/ does not match \"%\" in char_parser(:alnum)"
]
escaped_char_parser
this parser helps to parse escaped characters, while one could de this quite easily with the following example, one notices
that in order to get just an escpaed character two combinators, map
and sequence
are needed
iex(11)> escaped_quote_parser = sequence([
...(11)> char_parser("\\"), char_parser()])
...(11)> |> map(&Enum.at(&1, 1))
...(11)> { parse_string(escaped_quote_parser, "\\\""), parse_string(escaped_quote_parser, "\\'") }
{ {:ok, "\""}, {:ok, "'"} }
Compare this to the provided escaped_char_parser
:
iex(12)> parse_string(escaped_char_parser(), "\\a")
{:ok, "a"}
We can also change the escape character
iex(13)> parse_string(escaped_char_parser("%"), "%a")
{:ok, "a"}
iex(14)> parse_string(escaped_char_parser("%"), "\\a")
{:error, "\\ not member of \"%\" in char_parser(\"%\") in escaped_char_parser(%)"}
And furthermore we can restrict the set of which characters are allowed to be escaped
iex(15)> parser = escaped_char_parser("\\", "escape only \\", "\\")
...(15)> { parse_string(parser, "\\\\"), parse_string(parser, "\\a") }
{ {:ok, "\\"}, {:error, "a not member of \"\\\\\" in char_parser(\"\\\\\") in escape only \\"} }
Parsing sequences of characters
Keywords: keywords_parser
Does pretty much what is expected ;)
iex(16)> kwd_parser = keywords_parser(["do", "else", "if"])
...(16)> ["do", "if", "for"]
...(16)> |> Enum.map(&parse_string(kwd_parser, &1))
[
ok: "do",
ok: "if",
error: "no alternative could be parsed in keywords_parser([\"do\", \"else\", \"if\"])"
]
Identifiers: ident_parser
An identifier is defined by a character class for its first character and a character class for its subsequent characters, so one could define it roughly as
sequence([
first_char_parser,
many(second_char_paser)])
And that is how the ident_parser
is actually defined
iex(17)> parse_string(ident_parser(), "hello_42")
{:ok, "hello_42"}
iex(18)> parse_string(ident_parser(), "42hello_world")
{:error, "Regex ~r/[[:alpha:]]/ does not match \"4\" in char_parser(:alpha) in sequence"}
In Lisp we prefer -
to _
, no problem
iex(19)> parse_string(ident_parser("Lisp Style", additional_chars: "-"), "hello-42")
{:ok, "hello-42"}
But even the parser for the first character and the following characters can be defined
iex(20)> register_parser = ident_parser(
...(20)> "Uppercase and digit",
...(20)> first_char_parser: char_parser(:upper),
...(20)> rest_char_parser: char_parser(:digit),
...(20)> additional_chars: nil,
...(20)> max_len: 2,
...(20)> min_len: 2)
...(20)> [
...(20)> parse_string(register_parser, "R2"),
...(20)> parse_string(register_parser, "X_"),
...(20)> parse_string(register_parser, "R12"),
...(20)> parse_string(register_parser, "a2"),
...(20)> parse_string(register_parser, "ab")
...(20)> ]
[
ok: "R2",
error: "string \"X\" length 1 under required minimum 2",
error: "string \"R12\" length 3 exceeds allowed 2",
error: "Regex ~r/[[:upper:]]/ does not match \"a\" in char_parser(:upper) in sequence",
error: "Regex ~r/[[:upper:]]/ does not match \"a\" in char_parser(:upper) in sequence"
]
Predefined Combinators
sequence
Takes a list of Parsers
, only parses if all of them parse subsequently on the given
input and return a list of the results of each parser.
Let us remimplement the keywords parser
iex(21)> if_parser = sequence([char_parser("i"), char_parser("f")])
...(21)> |> map(&Enum.join/1)
...(21)> ~w[if else]
...(21)> |> Enum.map(&parse_string(if_parser, &1))
[
ok: "if",
error: "e not member of \"i\" in char_parser(\"i\") in sequence"
]
This leads us directly to
map
Map, takes a parser and a mapping function. It returns a new parser that fails with exactly the same error message as its input parser, but succeeds with the result mapped by the mapping function.
iex(22)> list_parser = many(char_parser()) |> map(&Enum.join(&1, ", "))
...(22)> parse_string(list_parser, "abc")
{:ok, "a, b, c"}
What about whitespace
Oftentimes whitespace shall be ignored in the resulting ast, and sometimes in the input too. To be more precise when whitespace stops parsing of example a keyword then the subsequent patser often is not interested in$ the left ofer ws preceeding its new input.
Enter ignore_ws
Here is the form that does not ignore newlines, which is the default:
iex(23)> next_char_parser = ignore_ws(char_parser())
...(23)> parse_string(next_char_parser, " \ta")
{:ok, "a"}
iex(24)> next_a_parser = ignore_ws(char_parser("a"))
...(24)> parse_string(next_a_parser, " \na")
{:error, "\n not member of \"a\" in char_parser(\"a\")"}
But we can also use the newline allowing version
iex(25)> next_a_parser = ignore_ws(char_parser("a"), "skip newlines", true)
...(25)> parse_string(next_a_parser, " \na")
{:ok, "a"}
upto_parser_parser
Oftentimes parsing algorithms become more read- and maintanable when we reparse a part of the
input stream with a different parser. In order to be able to do this we can just parse up to
a part of the input stream defined by a parser and return the input stream up to that point
as a String
N.B. This convenience comes with a price, the parser
will try to match for every position
in the input stream until it succeeds or fails on an empty input. Hence use with care.
iex(26)> upto_end_parser = upto_parser_parser(keywords_parser(~W[end]))
...(26)> parse_string(upto_end_parser, "up to end")
{:ok, "up to "}
If however parser
never succeeds the upto_parser_parser
fails.
iex(27)> upto_end_parser = upto_parser_parser(keywords_parser(~W[end]))
...(27)> parse_string(upto_end_parser, "up to en")
{:error, "encountered end of input in upto_parser_parser(keywords_parser([\"end\"]), keep)"}
We can also ask to include the ast from the parser
into the result
iex(28)> upto_end_parser = upto_parser_parser(keywords_parser(~W[end]), "my parser", :include)
...(28)> parse_string(upto_end_parser, "up to end")
{:ok, {"up to ", "end"}}
Or to discard it, which is not the default case (which is :keep
)
iex(29)> keep_parser = upto_parser_parser(keywords_parser(~W[end]), "my parser", :keep)
...(29)> discard_parser = upto_parser_parser(keywords_parser(~W[end]), "my other parser", :discard)
...(29)> [ parse(keep_parser, "up to end"), parse(discard_parser, "up to end")]
[
%Minipeg.Success{ast: "up to ", cache: %Minipeg.Cache{cache: %{}}, rest: %Minipeg.Input{chars: ["e", "n", "d"], col: 7, lnb: 1}},
%Minipeg.Success{ast: "up to ", cache: %Minipeg.Cache{cache: %{}}, rest: %Minipeg.Input{chars: [], col: 10, lnb: 1}}
]
Definitions
Parser
A Parser
is a struct that parses an Input
struct (with the parse
function) and either returns a Success
or Failure
struct
In order to abstract the internal representations of input and results the parse_string
function is provided as shown in the exampleas above.
The Success
struct contains the resulting Abstract Syntaxt Tree and the rest of the input as an Input
struct.
The Failure
struct contains the original Input
struct and an error message
Internally a Cache
is already returned (and passed into subsequent parse
calls of the Parser
module) but unless
you are extending Minipeg
itself by defining parsers by hand instead of using Combinators
you can ignore this.
Combinator
A Combibator
is a function that takes a Parser
optionally some arguments and returns a new Parser