View Source Tokenizers.Model.Unigram (Tokenizers v0.5.0)
Summary
Types
Options for model initialisation.
:byte_fallback
- whether to use the byte fallback trick:unk_id
- the unknown token id to be used by the model
Functions
@spec empty() :: {:ok, Tokenizers.Model.t()}
Instantiate an empty Unigram model
@spec init([{String.t(), number()}], options()) :: {:ok, Tokenizers.Model.t()}
Instantiate a Unigram model from the given vocab.