View Source tflite_beam_full_tokenizer (tflite_beam v0.3.5)

Runs end-to-end tokenization.

Related link: https://github.com/tensorflow/examples/blob/master/lite/examples/bert_qa/ios/BertQACore/Models/Tokenizers/FullTokenizer.swift

Link to this section Summary

Functions

Convert to ID in the vocab
End-to-end tokenization.

Link to this section Functions

Link to this function

convert_to_id(Tokens, Vocab)

View Source
-spec convert_to_id([binary()], map()) -> {ok, [integer()]} | {error, binary()}.
Convert to ID in the vocab
Link to this function

tokenize(Text, IsCaseInsensitive, Vocab)

View Source
-spec tokenize(binary() | list(), boolean(), map()) -> [binary()].
End-to-end tokenization.