Token estimation utilities for conversation context management.
Splits out the token-counting heuristics from Candil.Conversation so
the conversation module remains focused on message lifecycle.
Not part of the public API — used only by Candil.Conversation.
Algorithm
Uses the standard 4-chars-per-token heuristic (ceil(byte_size/4))
which is fast and works well for English/Code. For multi-lingual
content, consider integrating a proper tokenizer (e.g., tiktoken).
Summary
Functions
Estimates token count for a raw text string.
Legacy 4-chars-per-token heuristic. Faster than estimate_content/1
but less accurate for short or non-English text. Kept for callers
that need the exact old behaviour.
Backwards-compatible alias for estimate_content/1.
Estimates token count for a conversation by summing all message tokens plus a buffer for the system prompt.
Estimates token count for a single message map.
Backwards-compatible alias for estimate_message/1.
Backwards-compatible alias for estimate_content/1 (legacy name).
Functions
@spec estimate_content(String.t()) :: non_neg_integer()
Estimates token count for a raw text string.
Uses a per-word approximation: each whitespace-separated word
contributes one token (rounded up for long words to account for
sub-word splits common in BPE tokenizers). On typical English/Code
text this is within ±10% of tiktoken's count.
iex> TokenEstimator.estimate_content("hello world")
2
iex> TokenEstimator.estimate_content("antidisestablishmentarianism")
2The legacy 4-chars-per-token heuristic is still available via
estimate_content_legacy/1.
@spec estimate_content_legacy(String.t()) :: non_neg_integer()
Legacy 4-chars-per-token heuristic. Faster than estimate_content/1
but less accurate for short or non-English text. Kept for callers
that need the exact old behaviour.
Backwards-compatible alias for estimate_content/1.
@spec estimate_conversation(map()) :: non_neg_integer()
Estimates token count for a conversation by summing all message tokens plus a buffer for the system prompt.
@spec estimate_message(map()) :: non_neg_integer()
Estimates token count for a single message map.
Backwards-compatible alias for estimate_message/1.
Backwards-compatible alias for estimate_content/1 (legacy name).