Essence.Tokenizer.split_with_punctuation
You're seeing just the function
split_with_punctuation
, go back to Essence.Tokenizer module for more information.
Link to this function
split_with_punctuation(text)
Specs
split_with_punctuation(String.t()) :: List.t()
Splits a given String into tokens on punctuation, and include the punctuation as a token. This method supports Unicode text.