ExLLM.Bumblebee.TokenCounter (ex_llm v0.5.0)
View SourceToken counting utilities for local models.
Provides accurate token counting when Bumblebee is available, with fallback to heuristic estimation.
Summary
Functions
Count tokens for a list of messages.
Count tokens for the given text using the specified model's tokenizer.
Functions
Count tokens for a list of messages.
Examples
messages = [
%{role: "user", content: "Hello"},
%{role: "assistant", content: "Hi there!"}
]
{:ok, count} = TokenCounter.count_messages(messages, model: "microsoft/phi-2")
Count tokens for the given text using the specified model's tokenizer.
Falls back to heuristic estimation if model is not loaded or Bumblebee is not available.
Examples
{:ok, count} = TokenCounter.count_tokens("Hello world", model: "microsoft/phi-2")
# => {:ok, 2}
# Fallback when model not loaded
{:ok, estimate} = TokenCounter.count_tokens("Hello world")
# => {:ok, 3}