BinClass.Tokenizer (BinClass v0.3.0)

Copy Markdown View Source

Trains tokenizers and manages the metadata required by classifier artifacts.

Summary

Functions

metadata(tokenizer)

Returns the vocabulary size and required special-token IDs for a tokenizer.

train(data, opts \\ [])

Trains a BPE tokenizer from an enumerable of strings.

Functions

metadata(tokenizer)

Returns the vocabulary size and required special-token IDs for a tokenizer.

train(data, opts \\ [])

Trains a BPE tokenizer from an enumerable of strings.

Options

  • :vocab_size - Positive maximum vocabulary size. It must be at least 5 so all required special tokens can be included. Defaults to 8000.
  • :normalizer - Text normalizer to apply before tokenization. Accepts :nfkc_lowercase (Unicode NFKC + lowercasing, default), :nfkc, :lowercase, :none, nil, or a custom Tokenizers.Normalizer struct. Defaults to :nfkc_lowercase.

The tokenizer always includes [UNK], [CLS], [SEP], [PAD], and [MASK] as special tokens.