View Source TFLiteElixir.Tokenizer.BasicTokenizer (tflite_elixir v0.3.3)

Runs basic tokenization such as punctuation spliting, lower casing.

Link to this section Summary

Functions

Tokenizes a piece of text.

Link to this section Functions

Link to this function

tokenize(text, is_case_insensitive)

View Source
@spec tokenize(String.t(), boolean()) :: [String.t()]

Tokenizes a piece of text.