Essence.Tokenizer.tokenize
You're seeing just the function
tokenize
, go back to Essence.Tokenizer module for more information.
Link to this function
tokenize(text)
Specs
tokenize(String.t()) :: List.t()
Splits a given String into tokens. A token is a sequence of characters to
be treated as a group. The tokenize
method will split on whitespace and
punctuation, treating words and punctutation as tokens, and removing whitespace.