ExLLM.Context (ex_llm v0.1.0)
View SourceContext management for LLM conversations.
Provides utilities for managing conversation context windows, including:
- Token counting and estimation
- Message truncation strategies
- Context window validation
- System prompt preservation
Summary
Functions
Get context window size for a specific model.
Find models with context windows above a certain size.
Check if messages fit within context window.
Get context window size for a model.
Get context statistics for messages.
List all models with their context windows.
Prepare messages for sending to LLM, managing context window.
Get statistics about message context usage.
Truncate messages to fit within token limit.
Validate that messages fit within a model's context window.
Functions
@spec context_window_size(String.t(), String.t()) :: non_neg_integer() | nil
Get context window size for a specific model.
Parameters
provider
- LLM provider name (string)model
- Model name
Returns
Context window size in tokens or nil if unknown.
Find models with context windows above a certain size.
Examples
# Find models with at least 100k token context
large_models = ExLLM.Context.find_models_by_context(min_tokens: 100_000)
@spec fits_context?( atom() | String.t(), String.t(), [ExLLM.Types.message()], keyword() ) :: {:ok, boolean()} | {:error, term()}
Check if messages fit within context window.
Parameters
provider
- LLM providermodel
- Model namemessages
- Messages to check
Returns
{:ok, true}
if fits, {:ok, false}
if not, {:error, reason}
if unknown model.
Examples
{:ok, fits} = ExLLM.Context.fits_context?(:anthropic, "claude-3-5-sonnet-20241022", messages)
@spec get_context_window(atom() | String.t(), String.t()) :: non_neg_integer() | nil
Get context window size for a model.
Examples
window = ExLLM.Context.get_context_window(:openai, "gpt-4")
# => 8192
@spec get_stats( [ExLLM.Types.message()], keyword() ) :: map()
Get context statistics for messages.
Returns
Map with message count, token usage, and percentages.
Examples
stats = ExLLM.Context.get_stats(messages, max_tokens: 4096)
# => %{
# message_count: 10,
# estimated_tokens: 1234,
# max_tokens: 4096,
# tokens_used_percentage: 30.1,
# tokens_remaining: 2362
# }
@spec list_model_contexts() :: [map()]
List all models with their context windows.
Examples
models = ExLLM.Context.list_model_contexts()
# => [
# %{provider: "anthropic", model: "claude-3-5-sonnet-20241022", context_window: 200000},
# ...
# ]
@spec prepare_messages( [ExLLM.Types.message()], keyword() ) :: [ExLLM.Types.message()]
Prepare messages for sending to LLM, managing context window.
Options
:max_tokens
- Maximum context window size (defaults to model's limit):system_prompt
- System prompt to prepend:strategy
- Truncation strategy (:sliding_window
,:smart
,:none
):preserve_system
- Always keep system messages (default: true):reserve_tokens
- Tokens to reserve for response (default: 500)
Examples
messages = [
%{role: "system", content: "You are helpful."},
%{role: "user", content: "Hello"},
%{role: "assistant", content: "Hi!"}
]
# Auto-detect context window for model
prepared = ExLLM.Context.prepare_messages(messages,
provider: :anthropic,
model: "claude-3-5-sonnet-20241022"
)
# Manual context limit
prepared = ExLLM.Context.prepare_messages(messages,
max_tokens: 4096,
strategy: :smart
)
@spec stats([ExLLM.Types.message()]) :: map()
Get statistics about message context usage.
Parameters
messages
- List of conversation messages
Returns
Map with detailed statistics.
@spec truncate_messages( [ExLLM.Types.message()], keyword() ) :: [ExLLM.Types.message()]
Truncate messages to fit within token limit.
Parameters
messages
- List of messagesoptions
- Options including:max_tokens
Returns
Truncated list of messages.
@spec validate_context( [ExLLM.Types.message()], keyword() ) :: {:ok, non_neg_integer()} | {:error, {:context_too_large, map()}}
Validate that messages fit within a model's context window.
Options
:provider
- LLM provider name:model
- Model name:max_tokens
- Override default context window
Returns
{:ok, token_count}
if valid, {:error, {:context_too_large, details}}
if too large.