Agentix.Tokenizer.Heuristic (Agentix v0.1.0)

Copy Markdown View Source

The default Agentix.Tokenizer: a byte-count estimate (≈ English-text token density of ~4 bytes/token) with a deliberate ~20% over-estimate so budgeting errs toward leaving headroom — under-counting risks a hard over-window failure, while over-counting only compacts a little early. A +1 floor keeps non-empty text from counting as zero. No dependency; rough by design — see Agentix.Tokenizer.

Summary

Functions

Estimates the token count of text as bytes × 3 ÷ 10 + 1.

Functions

count(text)

Estimates the token count of text as bytes × 3 ÷ 10 + 1.

iex> Agentix.Tokenizer.Heuristic.count("hello world")
4