Penelope v0.1.0 Penelope.NLP.Tokenizer behaviour

The behaviour implemented by all tokenizers.

Link to this section Summary

Link to this section Callbacks

Link to this callback tokenize(arg0)
tokenize(String.t) :: [String.t]