View Source Tokenizers.Model.WordLevel (Tokenizers v0.5.0)
Summary
Functions
Instantiate an empty WordLevel model.
Instantiate a WordLevel model from the given vocab file.
Instantiate a WordLevel model from the given vocab.
Types
@type options() :: [{:unk_token, String.t()}]
Options for model initialisation.
:unk_token
- the unknown token to be used by the model. Defaults to "[UNK]"
Functions
@spec empty() :: {:ok, Tokenizers.Model.t()}
Instantiate an empty WordLevel model.
@spec from_file(String.t(), options()) :: {:ok, Tokenizers.Model.t()}
Instantiate a WordLevel model from the given vocab file.
@spec init( vocab :: %{required(String.t()) => integer()}, options :: options() ) :: {:ok, Tokenizers.Model.t()}
Instantiate a WordLevel model from the given vocab.