View Source tflite_beam_basic_tokenizer (tflite_beam v0.3.5)

Runs basic tokenization such as punctuation spliting, lower casing.

Related link: https://github.com/tensorflow/examples/blob/master/lite/examples/bert_qa/ios/BertQACore/Models/Tokenizers/BasicTokenizer.swift

Link to this section Summary

Link to this section Functions

Link to this function

split_by_whitespace(BinaryText)

View Source
Link to this function

tokenize(Text, IsCaseInsensitive)

View Source
-spec tokenize(binary() | list(), boolean()) -> [binary()].
Tokenizes a piece of text.