ExLLM.Context (ex_llm v0.5.0)

View Source

Context management for LLM conversations.

Provides utilities for managing conversation context windows, including:

  • Token counting and estimation
  • Message truncation strategies
  • Context window validation
  • System prompt preservation

Summary

Functions

Get the context window size for a given provider and model.

Get optimal token allocation for different message types.

Truncate messages to fit within context window.

Check if messages fit within the context window for a given model.

Functions

get_context_window(provider, model)

@spec get_context_window(String.t() | atom(), String.t()) :: pos_integer()

Get the context window size for a given provider and model.

Returns the maximum number of tokens the model can handle. Raises an error if the model is not found in configuration.

get_token_allocation(provider, model, options \\ [])

@spec get_token_allocation(String.t() | atom(), String.t(), keyword()) :: %{
  system: non_neg_integer(),
  conversation: non_neg_integer(),
  response: non_neg_integer(),
  total: non_neg_integer()
}

Get optimal token allocation for different message types.

Returns a map with recommended token allocations.

truncate_messages(messages, provider, model, options \\ [])

@spec truncate_messages(list(), String.t() | atom(), String.t(), keyword()) :: list()

Truncate messages to fit within context window.

Supports different truncation strategies:

  • :sliding_window - Remove old messages from the beginning
  • :smart - Preserve system message and recent messages, remove from middle

validate_context(messages, provider, model, options \\ [])

@spec validate_context(list(), String.t() | atom(), String.t(), keyword()) ::
  {:ok, non_neg_integer()} | {:error, String.t()}

Check if messages fit within the context window for a given model.

Returns {:ok, token_count} if messages fit, or {:error, reason} if not.