View Source TFLiteElixir.Tokenizer.FullTokenizer (tflite_elixir v0.4.0-rc3)

Runs end-to-end tokenization.

Related link: FullTokenizer.swift

Summary

Functions

convert_to_id(tokens, vocab)

Convert to ID in the vocab

tokenize(text, is_case_insensitive, vocab)

End-to-end tokenization.

Functions

Link to this function

convert_to_id(tokens, vocab)

View Source 
@spec convert_to_id([String.t()], map()) :: {:ok, [integer()]} | {:error, String.t()}

Convert to ID in the vocab

Link to this function

tokenize(text, is_case_insensitive, vocab)

View Source 
@spec tokenize(String.t(), boolean(), map()) :: [String.t()]

End-to-end tokenization.