LemonAi.
View Source
Fast token estimation helpers.
These functions centralize Lemon's rough 4-units-per-token heuristic. It is useful for thresholds, diagnostics, and fallback estimates, but it is not a tokenizer and will differ from model-specific token counts.
Use estimate_chars/1 when the caller is budgeting user-visible text by
Unicode character count. Use estimate_bytes/1 when the existing boundary is
payload bytes, serialized data, or transport size.
Summary
Functions
Estimates tokens from a precomputed byte count.
Estimates tokens from byte size using the 4 bytes/token heuristic.
Estimates tokens from a precomputed Unicode character count.
Estimates tokens from Unicode character count using the 4 chars/token heuristic.
Functions
@spec estimate_byte_count(non_neg_integer()) :: non_neg_integer()
Estimates tokens from a precomputed byte count.
@spec estimate_bytes(binary()) :: non_neg_integer()
Estimates tokens from byte size using the 4 bytes/token heuristic.
@spec estimate_char_count(non_neg_integer()) :: non_neg_integer()
Estimates tokens from a precomputed Unicode character count.
@spec estimate_chars(String.t()) :: non_neg_integer()
Estimates tokens from Unicode character count using the 4 chars/token heuristic.