View Source API Reference Tokenizers v0.4.0
Modules
Elixir bindings to Hugging Face Tokenizers.
This struct represents a token added to tokenizer vocabulary.
Decoders and decoding functions.
Encoding is the result of passing a text through tokenization pipeline.
Module containing handy functions to build the transformations list.
A simple implementation of an HTTP client.
The struct and associated functions for the tokenizer model.
Normalizers and normalization functions.
Post-processors.
Pre-tokenizers.
Functions to load, apply and train tokenizers.
A Trainer has the responsibility to train a model. We feed it with lines/sentences and then it can train the given Model.