View Source Tokenizers.Model.WordLevel (Tokenizers v0.5.1)

Summary

Types

Options for model initialisation.

Functions

Instantiate an empty WordLevel model.

Instantiate a WordLevel model from the given vocab file.

Instantiate a WordLevel model from the given vocab.

Types

@type options() :: [{:unk_token, String.t()}]

Options for model initialisation.

  • :unk_token - the unknown token to be used by the model. Defaults to "[UNK]"

Functions

@spec empty() :: {:ok, Tokenizers.Model.t()}

Instantiate an empty WordLevel model.

Link to this function

from_file(vocab_path, options \\ [])

View Source
@spec from_file(String.t(), options()) :: {:ok, Tokenizers.Model.t()}

Instantiate a WordLevel model from the given vocab file.

Link to this function

init(vocab, options \\ [])

View Source
@spec init(
  vocab :: %{required(String.t()) => integer()},
  options :: options()
) :: {:ok, Tokenizers.Model.t()}

Instantiate a WordLevel model from the given vocab.