Normandy.Context.Summarizer (normandy v2.2.0)

View Source

Handles conversation summarization for context window management.

When conversations exceed token limits, this module can summarize older messages to preserve context while reducing token usage.

Example

# Summarize old messages
{:ok, summary} = Summarizer.summarize_messages(client, agent, messages)

# Replace old messages with summary
{:ok, updated_agent} = Summarizer.compress_conversation(client, agent, keep_recent: 5)

Summary

Functions

compress_conversation(client, agent, opts \\ [])

Compresses a conversation by summarizing old messages.

estimate_savings(messages, opts \\ [])

Estimates token savings from summarization.

summarize_messages(client, agent, messages, opts \\ [])

Summarizes a list of messages using the LLM.

Functions

compress_conversation(client, agent, opts \\ [])

@spec compress_conversation(struct(), struct(), keyword()) ::
  {:ok, struct()} | {:error, term()}

Compresses a conversation by summarizing old messages.

Keeps recent messages intact and replaces older messages with a summary.

Options

  • :keep_recent (non_neg_integer/0) - Recent messages kept as they are. The kept messages start at a user message: the oldest one among the last :keep_recent, or the newest one when none is. The default value is 10.

  • :summary_role - The summary message's role. Not "system": the adapter sends the agent's own system prompt and refuses a second one. The default value is "user".

  • :prompt (String.t/0) - The summarization prompt (default: the built-in prompt, which keeps stated facts).

  • :model (String.t/0) - The model to summarize with (default: the agent's model).

  • :max_tokens (pos_integer/0) - Maximum tokens for the summary call (default: the agent's max_tokens, else 16_000). Thinking shares this budget, and current models cannot turn it off.

Example

{:ok, updated_agent} = Summarizer.compress_conversation(
  client,
  agent,
  keep_recent: 5
)

estimate_savings(messages, opts \\ [])

@spec estimate_savings(
  list(),
  keyword()
) :: {:ok, map()}

Estimates token savings from summarization.

Options

  • :summary_tokens (non_neg_integer/0) - The summary's expected size in tokens. The default value is 500.

Example

{:ok, savings} = Summarizer.estimate_savings(messages, summary_tokens: 200)
#=> %{original: 1500, summary: 200, savings: 1300, savings_percent: 86.7}

summarize_messages(client, agent, messages, opts \\ [])

@spec summarize_messages(struct(), struct(), list(), keyword()) ::
  {:ok, String.t()} | {:error, term()}

Summarizes a list of messages using the LLM.

Options

  • :prompt (String.t/0) - The summarization prompt (default: the built-in prompt, which keeps stated facts).

  • :model (String.t/0) - The model to summarize with (default: the agent's model).

  • :max_tokens (pos_integer/0) - Maximum tokens for the summary call (default: the agent's max_tokens, else 16_000). Thinking shares this budget, and current models cannot turn it off.

Example

messages = [
  %{role: "user", content: "Hello"},
  %{role: "assistant", content: "Hi there!"}
]

{:ok, summary} = Summarizer.summarize_messages(client, agent, messages)
#=> "User greeted, assistant responded"