View Source TFLiteElixir.Tokenizer.BasicTokenizer (tflite_elixir v0.3.3)
Runs basic tokenization such as punctuation spliting, lower casing.
Link to this section Summary
Functions
Tokenizes a piece of text.
Link to this section Functions
Tokenizes a piece of text.