ExLLM.Context (ex_llm v0.1.0)

View Source

Context management for LLM conversations.

Provides utilities for managing conversation context windows, including:

  • Token counting and estimation
  • Message truncation strategies
  • Context window validation
  • System prompt preservation

Summary

Functions

Get context window size for a specific model.

Find models with context windows above a certain size.

Check if messages fit within context window.

Get context window size for a model.

Get context statistics for messages.

List all models with their context windows.

Prepare messages for sending to LLM, managing context window.

Get statistics about message context usage.

Truncate messages to fit within token limit.

Validate that messages fit within a model's context window.

Functions

context_window_size(provider, model)

@spec context_window_size(String.t(), String.t()) :: non_neg_integer() | nil

Get context window size for a specific model.

Parameters

  • provider - LLM provider name (string)
  • model - Model name

Returns

Context window size in tokens or nil if unknown.

find_models_by_context(options \\ [])

@spec find_models_by_context(keyword()) :: [map()]

Find models with context windows above a certain size.

Examples

# Find models with at least 100k token context
large_models = ExLLM.Context.find_models_by_context(min_tokens: 100_000)

fits_context?(provider, model, messages, options \\ [])

@spec fits_context?(
  atom() | String.t(),
  String.t(),
  [ExLLM.Types.message()],
  keyword()
) ::
  {:ok, boolean()} | {:error, term()}

Check if messages fit within context window.

Parameters

  • provider - LLM provider
  • model - Model name
  • messages - Messages to check

Returns

{:ok, true} if fits, {:ok, false} if not, {:error, reason} if unknown model.

Examples

{:ok, fits} = ExLLM.Context.fits_context?(:anthropic, "claude-3-5-sonnet-20241022", messages)

get_context_window(provider, model)

@spec get_context_window(atom() | String.t(), String.t()) :: non_neg_integer() | nil

Get context window size for a model.

Examples

window = ExLLM.Context.get_context_window(:openai, "gpt-4")
# => 8192

get_stats(messages, options \\ [])

@spec get_stats(
  [ExLLM.Types.message()],
  keyword()
) :: map()

Get context statistics for messages.

Returns

Map with message count, token usage, and percentages.

Examples

stats = ExLLM.Context.get_stats(messages, max_tokens: 4096)
# => %{
#   message_count: 10,
#   estimated_tokens: 1234,
#   max_tokens: 4096,
#   tokens_used_percentage: 30.1,
#   tokens_remaining: 2362
# }

list_model_contexts()

@spec list_model_contexts() :: [map()]

List all models with their context windows.

Examples

models = ExLLM.Context.list_model_contexts()
# => [
#   %{provider: "anthropic", model: "claude-3-5-sonnet-20241022", context_window: 200000},
#   ...
# ]

prepare_messages(messages, options \\ [])

@spec prepare_messages(
  [ExLLM.Types.message()],
  keyword()
) :: [ExLLM.Types.message()]

Prepare messages for sending to LLM, managing context window.

Options

  • :max_tokens - Maximum context window size (defaults to model's limit)
  • :system_prompt - System prompt to prepend
  • :strategy - Truncation strategy (:sliding_window, :smart, :none)
  • :preserve_system - Always keep system messages (default: true)
  • :reserve_tokens - Tokens to reserve for response (default: 500)

Examples

messages = [
  %{role: "system", content: "You are helpful."},
  %{role: "user", content: "Hello"},
  %{role: "assistant", content: "Hi!"}
]

# Auto-detect context window for model
prepared = ExLLM.Context.prepare_messages(messages, 
  provider: :anthropic,
  model: "claude-3-5-sonnet-20241022"
)

# Manual context limit
prepared = ExLLM.Context.prepare_messages(messages,
  max_tokens: 4096,
  strategy: :smart
)

stats(messages)

@spec stats([ExLLM.Types.message()]) :: map()

Get statistics about message context usage.

Parameters

  • messages - List of conversation messages

Returns

Map with detailed statistics.

truncate_messages(messages, options \\ [])

@spec truncate_messages(
  [ExLLM.Types.message()],
  keyword()
) :: [ExLLM.Types.message()]

Truncate messages to fit within token limit.

Parameters

  • messages - List of messages
  • options - Options including :max_tokens

Returns

Truncated list of messages.

validate_context(messages, options \\ [])

@spec validate_context(
  [ExLLM.Types.message()],
  keyword()
) :: {:ok, non_neg_integer()} | {:error, {:context_too_large, map()}}

Validate that messages fit within a model's context window.

Options

  • :provider - LLM provider name
  • :model - Model name
  • :max_tokens - Override default context window

Returns

{:ok, token_count} if valid, {:error, {:context_too_large, details}} if too large.