ExLLM.Core.Context (ex_llm v0.8.1)
View SourceContext management for LLM conversations.
Provides utilities for managing conversation context windows, including:
- Token counting and estimation
- Message truncation strategies
- Context window validation
- System prompt preservation
Summary
Functions
Get the context window size for a given provider and model.
Get optimal token allocation for different message types.
Truncate messages to fit within context window.
Check if messages fit within the context window for a given model.
Functions
@spec get_context_window(String.t() | atom(), String.t()) :: pos_integer()
Get the context window size for a given provider and model.
Returns the maximum number of tokens the model can handle. Raises an error if the model is not found in configuration.
@spec get_token_allocation(String.t() | atom(), String.t(), keyword()) :: %{ system: non_neg_integer(), conversation: non_neg_integer(), response: non_neg_integer(), total: non_neg_integer() }
Get optimal token allocation for different message types.
Returns a map with recommended token allocations.
Truncate messages to fit within context window.
Supports different truncation strategies:
:sliding_window
- Remove old messages from the beginning:smart
- Preserve system message and recent messages, remove from middle
@spec validate_context(list(), String.t() | atom(), String.t(), keyword()) :: {:ok, non_neg_integer()} | {:error, String.t()}
Check if messages fit within the context window for a given model.
Returns {:ok, token_count}
if messages fit, or {:error, reason}
if not.