View Source Grammar.Tokenizer.TokenExtractor protocol (Grammar v0.3.0)
This protocol exposes the functions needed to extract tokens from the input string.
Summary
Functions
Returns true if the token matches the token prototype.
Try to read a token from the input string.
Types
@type t() :: term()
All the types that implement this protocol.
Functions
Returns true if the token matches the token prototype.
This function is used orient the parser in the right clause of a rule, by comparing the current token to the clause list of first tokens.
Example
iex> Grammar.Tokenizer.TokenExtractor.match?("hello", "hello")
true
iex> Grammar.Tokenizer.TokenExtractor.match?("hello", "Horld")
false
iex> Grammar.Tokenizer.TokenExtractor.match?(~r/[0-9]+/, "013456")
true
iex> Grammar.Tokenizer.TokenExtractor.match?(~r/[0-9]+/, "a013456")
false
Try to read a token from the input string.
If the token is found, returns the token and its length in the input string. Returns nil otherwise.
Example
iex> Grammar.Tokenizer.TokenExtractor.try_read("hello", "hello world")
{"hello", 5}
iex> Grammar.Tokenizer.TokenExtractor.try_read("Hello", "hello world")
nil
iex> Grammar.Tokenizer.TokenExtractor.try_read(~r/[0-9]+/, "013456toto")
{"013456", 6}
iex> Grammar.Tokenizer.TokenExtractor.try_read(~r/[a-z]+/, "013456toto")
nil