Search
tiktoken
v0.4.0
Pages
Modules
Settings
View Source
Tiktoken
(tiktoken v0.4.0)
Documentation for
Tiktoken
.
Link to this section
Summary
Functions
context_size_for_model(model)
decode(model, ids)
encode(model, text, allowed_special \\ [])
encode_ordinary(model, text)
encode_with_special_tokens(model, text)
encoding_for_model(model)
Link to this section
Functions
Link to this function
context_size_for_model(model)
View Source
Link to this function
decode(model, ids)
View Source
Link to this function
encode(model, text, allowed_special \\ [])
View Source
Link to this function
encode_ordinary(model, text)
View Source
Link to this function
encode_with_special_tokens(model, text)
View Source
Link to this function
encoding_for_model(model)
View Source