View Source Tokenizers.Model.Unigram (Tokenizers v0.5.1)

Summary

Types

Options for model initialisation.

Functions

Instantiate an empty Unigram model

Instantiate a Unigram model from the given vocab.

Types

@type options() :: [byte_fallback: boolean(), unk_id: integer()]

Options for model initialisation.

  • :byte_fallback- whether to use the byte fallback trick
  • :unk_id- the unknown token id to be used by the model

Functions

@spec empty() :: {:ok, Tokenizers.Model.t()}

Instantiate an empty Unigram model

Link to this function

init(vocab, options \\ [])

View Source
@spec init([{String.t(), number()}], options()) :: {:ok, Tokenizers.Model.t()}

Instantiate a Unigram model from the given vocab.