View Source TFLiteElixir.Tokenizer.BasicTokenizer (tflite_elixir v1.0.0)

Runs basic tokenization such as punctuation spliting, lower casing.

Summary

Functions

split_by_whitespace(text)

Split text on whitespace, dropping the runs between words.

tokenize(text, is_case_insensitive)

Tokenizes a piece of text.

Functions

Link to this function

split_by_whitespace(text)

View Source 
@spec split_by_whitespace(binary()) :: [String.t()]

Split text on whitespace, dropping the runs between words.

Link to this function

tokenize(text, is_case_insensitive)

View Source 
@spec tokenize(String.t(), boolean()) :: [String.t()]

Tokenizes a piece of text.