Essence.Tokenizer.tokenize

You're seeing just the function tokenize, go back to Essence.Tokenizer module for more information.

Specs

tokenize(String.t()) :: List.t()

Splits a given String into tokens. A token is a sequence of characters to be treated as a group. The tokenize method will split on whitespace and punctuation, treating words and punctutation as tokens, and removing whitespace.