LemonAi.ContextCompactor (lemon_ai v0.1.0)

View Source

Automatic context compaction for handling ContextLengthExceeded errors.

When an AI provider call fails due to exceeding the model's context window, this module provides strategies to compact the conversation context and enable automatic retry.

Strategies

  • :truncation - Remove oldest messages while preserving critical context
  • :summarization - Summarize older messages using a lightweight model
  • :hybrid - Combine truncation for recent history with summarization for older messages

Usage

# Check if error is a context length error
if LemonAi.ContextCompactor.context_length_error?(error) do
  # Compact the context
  {:ok, compacted_context} = LemonAi.ContextCompactor.compact(context, strategy: :truncation)
  # Retry with compacted context
end

Configuration

Configure via application environment:

config :lemon_ai, LemonAi.ContextCompactor,
  enabled: true,
  default_strategy: :truncation,
  max_compaction_attempts: 3,
  preserve_recent_messages: 4,
  min_context_tokens: 1000

Telemetry

The following telemetry events are emitted:

  • [:lemon_ai, :context_compactor, :compaction_started] - When compaction begins
  • [:lemon_ai, :context_compactor, :compaction_succeeded] - When compaction succeeds
  • [:lemon_ai, :context_compactor, :compaction_failed] - When compaction fails

Summary

Types

compaction_opts()
strategy()

Functions

compact(context, opts \\ [])

Compact a context using the specified strategy.

context_length_error?(error)

Check if an error represents a context length exceeded error.

default_strategy()

Get the default compaction strategy from configuration.

enabled?()

Check if compaction is enabled in configuration.

max_attempts()

Get the maximum number of compaction attempts from configuration.

Types

compaction_opts()

@type compaction_opts() :: [
  strategy: strategy(),
  preserve_recent: non_neg_integer(),
  target_tokens: non_neg_integer(),
  summarizer_model: String.t()
]

strategy()

@type strategy() :: :truncation | :summarization | :hybrid

Functions

compact(context, opts \\ [])

@spec compact(LemonAi.Types.Context.t(), compaction_opts()) ::
  {:ok, LemonAi.Types.Context.t(), map()} | {:error, term()}

Compact a context using the specified strategy.

Returns {:ok, compacted_context, metadata} on success, or {:error, reason} if compaction fails or would violate safety limits.

Options

  • :strategy - Compaction strategy (:truncation, :summarization, :hybrid)
  • :preserve_recent - Number of recent messages to always preserve (default: 4)
  • :target_tokens - Target token count after compaction (optional)
  • :summarizer_model - Model to use for summarization (default: "gpt-4o-mini")

Examples

{:ok, compacted, metadata} = LemonAi.ContextCompactor.compact(context, strategy: :truncation)

context_length_error?(error)

@spec context_length_error?(term()) :: boolean()

Check if an error represents a context length exceeded error.

Detects context length errors from various provider response formats.

Examples

iex> LemonAi.ContextCompactor.context_length_error?({:http_error, 400, %{"error" => %{"code" => "context_length_exceeded"}}})
true

iex> LemonAi.ContextCompactor.context_length_error?({:http_error, 429, "Rate limited"})
false

default_strategy()

@spec default_strategy() :: strategy()

Get the default compaction strategy from configuration.

enabled?()

@spec enabled?() :: boolean()

Check if compaction is enabled in configuration.

max_attempts()

@spec max_attempts() :: non_neg_integer()

Get the maximum number of compaction attempts from configuration.