View Source Tokenizers.Model.Unigram (Tokenizers v0.4.0)

Summary

Types

Options for model initialisation.

Functions

Instantiate an empty Unigram model

Instantiate a Unigram model from the given vocab.

Types

@type options() :: [{:unk_id, float()}]

Options for model initialisation.

  • :unk_id- the unknown token id to be used by the model

Functions

@spec empty() :: {:ok, Tokenizers.Model.t()}

Instantiate an empty Unigram model

Link to this function

init(vocab, options \\ [])

View Source
@spec init([{String.t(), number()}], options()) :: {:ok, Tokenizers.Model.t()}

Instantiate a Unigram model from the given vocab.