Toggle Sidebar
Penelope
v0.1.0
Pages
Modules
Exceptions
Mix Tasks
Penelope v0.1.0
Penelope.NLP.Tokenizer
behaviour
The behaviour implemented by all tokenizers.
Link to this section
Summary
Callbacks
tokenize(arg0)
Link to this section
Callbacks
Link to this callback
tokenize(arg0)
tokenize(
String.t
) :: [
String.t
]