ExLLM.Bumblebee.TokenCounter (ex_llm v0.5.0)

View Source

Token counting utilities for local models.

Provides accurate token counting when Bumblebee is available, with fallback to heuristic estimation.

Summary

Functions

Count tokens for a list of messages.

Count tokens for the given text using the specified model's tokenizer.

Functions

count_messages(messages, opts \\ [])

Count tokens for a list of messages.

Examples

messages = [
  %{role: "user", content: "Hello"},
  %{role: "assistant", content: "Hi there!"}
]
{:ok, count} = TokenCounter.count_messages(messages, model: "microsoft/phi-2")

count_tokens(text, opts \\ [])

Count tokens for the given text using the specified model's tokenizer.

Falls back to heuristic estimation if model is not loaded or Bumblebee is not available.

Examples

{:ok, count} = TokenCounter.count_tokens("Hello world", model: "microsoft/phi-2")
# => {:ok, 2}

# Fallback when model not loaded
{:ok, estimate} = TokenCounter.count_tokens("Hello world")
# => {:ok, 3}