Essence.Tokenizer.split_without_punctuation
You're seeing just the function
split_without_punctuation
, go back to Essence.Tokenizer module for more information.
Link to this function
split_without_punctuation(text)
Specs
split_without_punctuation(String.t()) :: List.t()
Splits a given text
into tokens on punctuation, but omits the punctuation tokens.
This method supports Unicode text.