fnord
v0.7.14
Search documentation of fnord
Settings
AI.Tokenizer.Tokens_o200k_base
(fnord v0.7.14)
View Source
This module implements the o200k-base tokenizer model used by OpenAI's
gpt-4o
models.
Summary
Functions
get_merges()
get_pattern()
get_reverse_vocab()
get_special_tokens()
get_vocab()
Functions
get_merges()
get_pattern()
get_reverse_vocab()
get_special_tokens()
get_vocab()