ExLLM.Core.Chat (ex_llm v0.8.1)

View Source

Core chat functionality for ExLLM.

This module contains the primary chat operations that power ExLLM's unified interface. It handles regular and streaming chat requests, with automatic context management, retry logic, and cost tracking.

Usage

While this module can be used directly, most users should use the main ExLLM module which delegates to these functions.

# Direct usage
{:ok, response} = ExLLM.Core.Chat.chat(:anthropic, messages, options)

# Recommended usage
{:ok, response} = ExLLM.chat(:anthropic, messages, options)

Features

  • Context Management: Automatic message truncation for model limits
  • Retry Logic: Configurable retry with exponential backoff
  • Cost Tracking: Automatic usage and cost calculation
  • Structured Output: Schema validation via instructor integration
  • Function Calling: Unified function calling across providers
  • Streaming: Real-time response streaming with recovery
  • Caching: Response caching for improved performance
  • Telemetry: Comprehensive instrumentation

Summary

Functions

Send a chat completion request to the specified LLM provider.

Send a streaming chat completion request to the specified LLM provider.

Types

messages()

@type messages() :: [ExLLM.Types.message()]

options()

@type options() :: keyword()

provider()

@type provider() :: ExLLM.provider()

Functions

chat(provider_or_model, messages, options \\ [])

@spec chat(provider() | String.t(), messages(), options()) ::
  {:ok, ExLLM.Types.LLMResponse.t() | struct() | map()} | {:error, term()}

Send a chat completion request to the specified LLM provider.

This is the core chat function that handles regular (non-streaming) requests. It includes context management, retry logic, cost tracking, and optional structured output via instructor integration.

Parameters

  • provider - The LLM provider (:anthropic, :openai, :groq, etc.) or a model string like "groq/llama3-70b"
  • messages - List of conversation messages
  • options - Options for the request

Options

See ExLLM.chat/3 for complete options documentation.

Returns

{:ok, %ExLLM.Types.LLMResponse{}} on success, or {:ok, struct} when using response_model. Returns {:error, reason} on failure.

stream_chat(provider_or_model, messages, options \\ [])

@spec stream_chat(provider() | String.t(), messages(), options()) ::
  {:ok, ExLLM.Types.stream()} | {:error, term()}

Send a streaming chat completion request to the specified LLM provider.

This function handles streaming requests with optional recovery capabilities. It provides real-time response chunks and can automatically recover from interruptions.

Parameters

  • provider - The LLM provider (:anthropic, :openai, :ollama)
  • messages - List of conversation messages
  • options - Options for the request

Options

Same as chat/3, plus:

  • :on_chunk - Callback function for each chunk
  • :stream_recovery - Enable automatic stream recovery (default: false)
  • :recovery_strategy - How to resume: :exact, :paragraph, or :summarize
  • :recovery_id - Custom ID for recovery (auto-generated if not provided)

Returns

{:ok, stream} on success where stream yields %ExLLM.Types.StreamChunk{} structs, {:error, reason} on failure.