LemonAi.CompactingClient (lemon_ai v0.1.0)

View Source

AI client with automatic context compaction and retry on ContextLengthExceeded.

This module wraps the standard AI provider calls with automatic retry logic that compacts the context when a ContextLengthExceeded error is encountered.

Usage

Use this module as a drop-in replacement for direct provider calls:

# Instead of:
LemonAi.Providers.Anthropic.stream(model, context, opts)

# Use:
LemonAi.CompactingClient.stream(model, context, opts)

The client will automatically:

  1. Attempt the initial request
  2. Detect ContextLengthExceeded errors
  3. Compact the context using the configured strategy
  4. Retry with the compacted context
  5. Repeat up to the maximum number of compaction attempts

Configuration

Configure via application environment:

config :lemon_ai, LemonAi.CompactingClient,
  enabled: true,
  max_compaction_attempts: 3,
  default_strategy: :truncation

Telemetry

  • [:lemon_ai, :compacting_client, :request_started] - When a request begins
  • [:lemon_ai, :compacting_client, :compaction_retry] - When compaction retry occurs
  • [:lemon_ai, :compacting_client, :request_succeeded] - When request succeeds
  • [:lemon_ai, :compacting_client, :request_failed] - When request fails permanently

Summary

Functions

enabled?()

Check if compaction is enabled globally.

max_attempts()

Get the maximum number of compaction attempts from configuration.

stream(model, context, opts \\ [])

Stream a response from the provider with automatic compaction retry.

Functions

enabled?()

@spec enabled?() :: boolean()

Check if compaction is enabled globally.

max_attempts()

@spec max_attempts() :: non_neg_integer()

Get the maximum number of compaction attempts from configuration.

stream(model, context, opts \\ [])

@spec stream(
  LemonAi.Types.Model.t(),
  LemonAi.Types.Context.t(),
  LemonAi.Types.StreamOptions.t() | keyword()
) :: {:ok, LemonAi.EventStream.t()} | {:error, term()}

Stream a response from the provider with automatic compaction retry.

This function wraps the provider's stream function and automatically retries with compacted context if a ContextLengthExceeded error occurs.

Options

All standard StreamOptions are supported, plus:

  • :compaction_enabled - Enable/disable compaction for this request (default: true)
  • :compaction_strategy - Strategy to use (:truncation, :summarization, :hybrid)
  • :max_compaction_attempts - Maximum number of compaction retries

Examples

{:ok, stream} = LemonAi.CompactingClient.stream(model, context, opts)
result = LemonAi.EventStream.result(stream)