Conversation history management for Candil.Llm.
Maintains a message history and automatically manages context window
limits. When the accumulated token estimate exceeds max_context_tokens,
older messages are trimmed while always preserving the system prompt.
Usage
conv = Candil.Conversation.new(
model: :llama3,
system: "You are a helpful Elixir assistant.",
max_context_tokens: 4096
)
{:ok, conv, response} = Candil.Conversation.chat(conv, "What is a GenServer?")
{:ok, conv, response} = Candil.Conversation.chat(conv, "Give me a code example.")
IO.puts(response.content)Remote provider
conv = Candil.Conversation.new(
model: gpt4o_model,
provider: openai_provider,
system: "You are a code reviewer.",
max_context_tokens: 16_000
)Token estimation
Token counts are estimated using a more accurate approximation that accounts for:
- Per-message overhead (role, content wrapper): ~4 tokens
- Per-message overhead for message arrays: ~3 tokens
- Content itself: approximately
ceil(byte_size / 4)for English text
The formula used is: 4 + ceil(content_bytes / 4) per message.
For more precise estimation, consider using a tokenizer library like tiktoken
if available for your model.
Configuration
The following options can be set in application config:
config :candil, Candil.Conversation,
max_tokens: 512, # default max_tokens for responses
estimation_mode: :default # :default or :tiktoken (when available)Summary
Functions
Returns the available context tokens (accounting for max_response_tokens).
Sends a user message and returns {:ok, updated_conv, response}.
Estimates tokens for text content.
Estimates tokens for a message, accounting for role and overhead.
Returns the full message list including the system prompt as the first message (if set).
Creates a new conversation.
Resets the conversation history, keeping the system prompt and config.
Returns the approximate token count for the current history.
Returns the number of turns (user+assistant pairs) in the conversation.
Types
@type message() :: Candil.Inference.message()
@type t() :: %Candil.Conversation{ max_context_tokens: pos_integer(), max_response_tokens: pos_integer(), messages: [message()], model: atom() | Candil.Model.t(), opts: keyword(), provider: Candil.Provider.t() | nil, system: binary() | nil }
Functions
@spec available_context_tokens(t()) :: non_neg_integer()
Returns the available context tokens (accounting for max_response_tokens).
@spec chat(t(), binary()) :: {:ok, t(), Candil.Inference.response()} | {:error, any()}
Sends a user message and returns {:ok, updated_conv, response}.
Appends the user message to history, calls the model, appends the assistant response, and trims history if needed.
@spec estimate_content_tokens(binary()) :: non_neg_integer()
Estimates tokens for text content.
Uses ceil(byte_size / 4) as a rough approximation for English text.
Note: This is a rough estimate. For precise counts, use a tokenizer
like tiktoken.
@spec estimate_message_tokens(message()) :: non_neg_integer()
Estimates tokens for a message, accounting for role and overhead.
The formula is:
- Base overhead per message: ~4 tokens
- Content:
ceil(byte_size / 4)for typical English text
Returns the full message list including the system prompt as the first message (if set).
Creates a new conversation.
Options
:model— atom alias (local engine) orCandil.Modelstruct (required):provider—Candil.Providerstruct for remote models:system— system prompt (default:nil):max_context_tokens— approximate token limit for history (default:4096):max_response_tokens— max tokens to generate in responses (default:512)- Any other options are forwarded to
Candil.chat/3on each turn (:temperature,:max_tokens, etc.)
Resets the conversation history, keeping the system prompt and config.
@spec token_estimate(t()) :: non_neg_integer()
Returns the approximate token count for the current history.
@spec turn_count(t()) :: non_neg_integer()
Returns the number of turns (user+assistant pairs) in the conversation.