View Source API Reference Tokenizers v0.5.1

Modules

Elixir bindings to Hugging Face Tokenizers.

This struct represents a token added to tokenizer vocabulary.

Decoders and decoding functions.

Encoding is the result of passing a text through tokenization pipeline.

Module containing handy functions to build the transformations list.

A simple implementation of an HTTP client.

The struct and associated functions for the tokenizer model.

Normalizers and normalization functions.

Post-processors.

Pre-tokenizers.

Functions to load, apply and train tokenizers.

A Trainer has the responsibility to train a model. We feed it with lines/sentences and then it can train the given Model.