View Source Tokenizers.Model.Unigram (Tokenizers v0.4.0)
Summary
Types
@type options() :: [{:unk_id, float()}]
Options for model initialisation.
:unk_id
- the unknown token id to be used by the model
Functions
@spec empty() :: {:ok, Tokenizers.Model.t()}
Instantiate an empty Unigram model
@spec init([{String.t(), number()}], options()) :: {:ok, Tokenizers.Model.t()}
Instantiate a Unigram model from the given vocab.