Tokenizers
v0.1.1
Pages
Modules
Settings
View Source
Tokenizers.Native
(Tokenizers v0.1.1)
Link to this section
Summary
Functions
decode(tokenizer, ids, skip_special_tokens)
decode_batch(tokenizer, ids, skip_special_tokens)
encode(tokenizer, input, add_special_tokens)
encode_batch(tokenizer, input, add_special_tokens)
from_file(path)
from_pretrained(identifier)
get_attention_mask(encoding)
get_ids(encoding)
get_model(tokenizer)
get_model_details(model)
get_offsets(encoding)
get_special_tokens_mask(encoding)
get_tokens(encoding)
get_type_ids(encoding)
get_vocab(tokenizer, with_added_tokens)
get_vocab_size(tokenizer, with_added_tokens)
id_to_token(tokenizer, id)
n_tokens(encoding)
pad(encoding, target_length, pad_id, pad_type_id, pad_token, direction)
save(tokenizer, path, pretty)
token_to_id(tokenizer, token)
truncate(encoding, max_len, stride, direction)
Link to this section
Functions
Link to this function
decode(tokenizer, ids, skip_special_tokens)
View Source
Link to this function
decode_batch(tokenizer, ids, skip_special_tokens)
View Source
Link to this function
encode(tokenizer, input, add_special_tokens)
View Source
Link to this function
encode_batch(tokenizer, input, add_special_tokens)
View Source
Link to this function
from_file(path)
View Source
Link to this function
from_pretrained(identifier)
View Source
Link to this function
get_attention_mask(encoding)
View Source
Link to this function
get_ids(encoding)
View Source
Link to this function
get_model(tokenizer)
View Source
Link to this function
get_model_details(model)
View Source
Link to this function
get_offsets(encoding)
View Source
Link to this function
get_special_tokens_mask(encoding)
View Source
Link to this function
get_tokens(encoding)
View Source
Link to this function
get_type_ids(encoding)
View Source
Link to this function
get_vocab(tokenizer, with_added_tokens)
View Source
Link to this function
get_vocab_size(tokenizer, with_added_tokens)
View Source
Link to this function
id_to_token(tokenizer, id)
View Source
Link to this function
n_tokens(encoding)
View Source
Link to this function
pad(encoding, target_length, pad_id, pad_type_id, pad_token, direction)
View Source
Link to this function
save(tokenizer, path, pretty)
View Source
Link to this function
token_to_id(tokenizer, token)
View Source
Link to this function
truncate(encoding, max_len, stride, direction)
View Source