ExLLM (ex_llm v0.5.0)

View Source

ExLLM - Unified Elixir client library for Large Language Models.

ExLLM provides a consistent interface across multiple LLM providers including OpenAI, Anthropic Claude, Ollama, and others. It features configuration injection, standardized error handling, and streaming support.

Quick Start

# Using environment variables
messages = [%{role: "user", content: "Hello!"}]
{:ok, response} = ExLLM.chat(:anthropic, messages)
IO.puts(response.content)

# Using static configuration
config = %{anthropic: %{api_key: "your-key"}}
{:ok, provider} = ExLLM.ConfigProvider.Static.start_link(config)
{:ok, response} = ExLLM.chat(:anthropic, messages, config_provider: provider)

Supported Providers

  • :anthropic - Anthropic Claude models
  • :openai - OpenAI GPT models
  • :groq - Groq (fast inference)
  • :lmstudio - Local models via LM Studio (desktop app)
  • :mistral - Mistral AI models (European AI)
  • :perplexity - Perplexity AI models (search-augmented)
  • :openrouter - OpenRouter (300+ models from multiple providers)
  • :ollama - Local models via Ollama
  • :bedrock - AWS Bedrock (multiple providers)
  • :gemini - Google Gemini models
  • :xai - X.AI Grok models
  • :bumblebee - Local models via Bumblebee (Phi-2, Llama 2, Mistral, etc.)
  • :mock - Mock adapter for testing

Features

  • Unified Interface: Same API across all providers
  • Configuration Injection: Flexible config management
  • Streaming Support: Real-time response streaming with error recovery
  • Error Standardization: Consistent error handling
  • Function Calling: Unified interface for tool use across providers
  • Model Discovery: Query and compare model capabilities
  • Automatic Retries: Exponential backoff with provider-specific policies
  • Mock Adapter: Built-in testing support without API calls
  • Cost Tracking: Automatic usage and cost calculation
  • Context Management: Automatic message truncation for model limits
  • Session Management: Conversation state tracking
  • Structured Outputs: Schema validation via instructor integration
  • No Process Dependencies: Pure functional core
  • Extensible: Easy to add new providers

Configuration

ExLLM supports multiple configuration methods:

Environment Variables

export ANTHROPIC_API_KEY="api-..."
export OPENAI_API_KEY="sk-..."
export GROQ_API_KEY="gsk-..."
export OPENROUTER_API_KEY="sk-or-..."
export OLLAMA_API_BASE="http://localhost:11434"
export GOOGLE_API_KEY="your-key"
export XAI_API_KEY="xai-..."
export AWS_ACCESS_KEY_ID="your-key"
export AWS_SECRET_ACCESS_KEY="your-secret"

Static Configuration

config = %{
  anthropic: %{api_key: "api-...", model: "claude-3-5-sonnet-20241022"},
  openai: %{api_key: "sk-...", model: "gpt-4"},
  openrouter: %{api_key: "sk-or-...", model: "openai/gpt-4o"},
  ollama: %{base_url: "http://localhost:11434", model: "llama2"},
  bedrock: %{access_key_id: "...", secret_access_key: "...", region: "us-east-1"},
  gemini: %{api_key: "...", model: "gemini-pro"},
  bumblebee: %{model: "microsoft/phi-2"}
}
{:ok, provider} = ExLLM.ConfigProvider.Static.start_link(config)

Custom Configuration

defmodule MyConfigProvider do
  @behaviour ExLLM.ConfigProvider

  def get([:anthropic, :api_key]), do: MyApp.get_secret("anthropic_key")
  def get(_), do: nil

  def get_all(), do: %{}
end

Examples

# Simple chat
{:ok, response} = ExLLM.chat(:anthropic, [
  %{role: "user", content: "What is Elixir?"}
])

# With options
{:ok, response} = ExLLM.chat(:anthropic, messages,
  model: "claude-3-haiku-20240307",
  temperature: 0.7,
  max_tokens: 1000
)

# Streaming
{:ok, stream} = ExLLM.stream_chat(:anthropic, messages)
for chunk <- stream do
  if chunk.content, do: IO.write(chunk.content)
end

# Check if provider is configured
if ExLLM.configured?(:anthropic) do
  {:ok, response} = ExLLM.chat(:anthropic, messages)
end

# List available models
{:ok, models} = ExLLM.list_models(:anthropic)
Enum.each(models, fn model ->
  IO.puts(model.name)
end)

Summary

Functions

Analyze images with a specific prompt.

Calculate cost for token usage.

Send a chat completion request to the specified LLM provider.

Send a chat request using a session, automatically tracking messages and usage.

Clear messages from a session while preserving metadata.

Compare capabilities across multiple models.

Compare capabilities across multiple providers.

Check if the specified provider is properly configured.

Get statistics about message context usage.

Get default model for a provider.

Calculate similarity between two embeddings.

Get the default model for the specified provider.

Generate embeddings for text inputs.

Estimate token count for text.

Execute a function call with available functions.

Extract text from an image using vision capabilities.

Find models that support specific features.

Find providers that support all specified features.

Find the most similar items from a list of embeddings.

Format cost for display.

Format function result for conversation continuation.

Get complete capability information for a model.

Get provider-level capabilities.

Get messages from a session.

Check if a provider is local (no API calls).

List available embedding models for a provider.

List all trackable model features.

List available models for the specified provider.

List all available providers.

List all recoverable streams.

Load an image from file for use in vision requests.

Load a session from JSON.

Check if a model supports a specific feature.

Get models grouped by a specific capability.

Create a new conversation session.

Parse function calls from an LLM response.

Prepare messages for sending to a provider with context management.

Check if a provider requires authentication.

Check if a provider supports a specific feature or endpoint.

Get recommended models based on requirements.

Get provider recommendations based on requirements.

Resume a previously interrupted stream.

Save a session to JSON.

Get total token usage for a session.

Send a streaming chat completion request to the specified LLM provider.

Get list of supported providers.

Check if a provider and model support vision inputs.

Validate that messages fit within a model's context window.

Create a vision-enabled message with text and images.

Types

messages()

@type messages() :: [ExLLM.Types.message()]

options()

@type options() :: keyword()

provider()

@type provider() ::
  :anthropic
  | :bumblebee
  | :openai
  | :groq
  | :lmstudio
  | :mistral
  | :perplexity
  | :openrouter
  | :ollama
  | :bedrock
  | :gemini
  | :xai
  | :mock

Functions

add_session_message(session, role, content, opts \\ [])

Add a message to a session.

Parameters

  • session - The session to update
  • role - Message role ("user", "assistant", etc.)
  • content - Message content
  • opts - Additional message metadata

Returns

Updated session.

Examples

session = ExLLM.add_session_message(session, "user", "What is Elixir?")

analyze_images(provider, image_paths, prompt, options \\ [])

@spec analyze_images(provider(), [String.t()], String.t(), keyword()) ::
  {:ok, String.t()} | {:error, term()}

Analyze images with a specific prompt.

Examples

{:ok, analysis} = ExLLM.analyze_images(:anthropic,
  ["chart1.png", "chart2.png"],
  "Compare these two charts and identify key differences"
)

calculate_cost(provider, model, token_usage)

@spec calculate_cost(provider(), String.t(), ExLLM.Types.token_usage()) ::
  ExLLM.Types.cost_result() | %{error: String.t()}

Calculate cost for token usage.

Parameters

  • provider - LLM provider name
  • model - Model name
  • token_usage - Map with :input_tokens and :output_tokens

Returns

Cost calculation result or error map.

Examples

usage = %{input_tokens: 1000, output_tokens: 500}
cost = ExLLM.calculate_cost("openai", "gpt-4", usage)
# => %{total_cost: 0.06, ...}

chat(provider_or_model, messages, options \\ [])

@spec chat(provider() | String.t(), messages(), options()) ::
  {:ok, ExLLM.Types.LLMResponse.t() | struct() | map()} | {:error, term()}

Send a chat completion request to the specified LLM provider.

Parameters

  • provider - The LLM provider (:anthropic, :openai, :groq, etc.) or a model string like "groq/llama3-70b"
  • messages - List of conversation messages
  • options - Options for the request (see module docs)

Options

  • :model - Override the default model
  • :temperature - Temperature setting (0.0 to 1.0)
  • :max_tokens - Maximum tokens in response or context
  • :config_provider - Configuration provider module or pid
  • :track_cost - Whether to track costs (default: true)
  • :strategy - Context truncation strategy (default: :sliding_window)
    • :sliding_window - Keep most recent messages
    • :smart - Preserve system messages and recent context
  • :preserve_messages - Number of recent messages to always preserve (default: 5)
  • :response_model - Ecto schema or type spec for structured output (requires instructor)
  • :max_retries - Number of retries for structured output validation
  • :functions - List of available functions for function calling
  • :function_call - Control function calling: "auto", "none", or specific function
  • :tools - Alternative to functions for providers that use tools API
  • :retry - Enable automatic retry (default: true)
  • :retry_count - Number of retry attempts (default: 3)
  • :retry_delay - Initial retry delay in ms (default: 1000)
  • :retry_backoff - Backoff strategy: :exponential or :linear (default: :exponential)
  • :retry_jitter - Add jitter to retry delays (default: true)
  • :stream_recovery - Enable stream recovery (default: false)
  • :recovery_strategy - Recovery strategy: :exact, :paragraph, or :summarize
  • :cache - Enable caching for this request (default: false unless globally enabled)
  • :cache_ttl - Cache TTL in milliseconds (default: 15 minutes)
  • :timeout - Request timeout in milliseconds (provider-specific defaults)
    • Ollama default: 120000 (2 minutes)
    • Other providers use their client library defaults
  • Mock adapter options:
    • :mock_response - Static response or response map
    • :mock_handler - Function to generate dynamic responses
    • :mock_error - Simulate an error
    • :mock_chunks - List of chunks for streaming
    • :chunk_delay - Delay between chunks in ms
    • :capture_requests - Capture requests for testing

Returns

{:ok, %ExLLM.Types.LLMResponse{}} on success, or {:ok, struct} when using response_model. Returns {:error, reason} on failure.

Examples

# Simple usage
{:ok, response} = ExLLM.chat(:anthropic, [
  %{role: "user", content: "Hello!"}
])

# Using provider/model syntax
{:ok, response} = ExLLM.chat("groq/llama3-70b", [
  %{role: "user", content: "Hello!"}
])

# Groq with specific model
{:ok, response} = ExLLM.chat(:groq, messages, model: "mixtral-8x7b-32768")

# With custom configuration
{:ok, provider} = ExLLM.ConfigProvider.Static.start_link(%{
  anthropic: %{api_key: "your-key"}
})
{:ok, response} = ExLLM.chat(:anthropic, messages, config_provider: provider)

# With model override
{:ok, response} = ExLLM.chat(:openai, messages, model: "gpt-4-turbo")

# With context management
{:ok, response} = ExLLM.chat(:anthropic, messages,
  max_tokens: 4000,
  strategy: :smart
)

# With structured output (requires instructor)
{:ok, classification} = ExLLM.chat(:anthropic, messages,
  response_model: EmailClassification,
  max_retries: 3
)

# With function calling
functions = [
  %{
    name: "get_weather",
    description: "Get current weather",
    parameters: %{
      type: "object",
      properties: %{location: %{type: "string"}},
      required: ["location"]
    }
  }
]

{:ok, response} = ExLLM.chat(:openai, messages,
  functions: functions,
  function_call: "auto"
)

chat_with_session(session, content, options \\ [])

@spec chat_with_session(ExLLM.Session.Types.Session.t(), String.t(), options()) ::
  {:ok, {ExLLM.Types.LLMResponse.t(), ExLLM.Session.Types.Session.t()}}
  | {:error, term()}

Send a chat request using a session, automatically tracking messages and usage.

Parameters

  • session - The session to use
  • content - The user message content
  • options - Chat options (same as chat/3)

Returns

{:ok, {response, updated_session}} on success, {:error, reason} on failure.

Examples

session = ExLLM.new_session(:anthropic)
{:ok, {response, session}} = ExLLM.chat_with_session(session, "Hello!")
# Session now contains the conversation history

clear_session(session)

Clear messages from a session while preserving metadata.

Parameters

  • session - The session to clear

Returns

Updated session with no messages.

Examples

session = ExLLM.clear_session(session)

compare_models(model_specs)

@spec compare_models([{provider(), String.t()}]) :: map()

Compare capabilities across multiple models.

Examples

comparison = ExLLM.compare_models([
  {:openai, "gpt-4-turbo"},
  {:anthropic, "claude-3-5-sonnet-20241022"},
  {:gemini, "gemini-pro"}
])

# See which features each model supports
comparison.features[:vision]
# => [%{supported: true}, %{supported: true}, %{supported: false}]

compare_providers(providers)

@spec compare_providers([provider()]) :: map()

Compare capabilities across multiple providers.

Examples

comparison = ExLLM.compare_providers([:openai, :anthropic, :ollama])

# See all features across providers
comparison.features
# => [:streaming, :function_calling, :vision, ...]

# Check specific provider capabilities
comparison.comparison.openai.features
# => [:streaming, :function_calling, :cost_tracking, ...]

configured?(provider, options \\ [])

@spec configured?(provider(), options()) :: boolean()

Check if the specified provider is properly configured.

Parameters

  • provider - The LLM provider to check
  • options - Options including configuration provider

Returns

true if configured, false otherwise.

Examples

if ExLLM.configured?(:anthropic) do
  {:ok, response} = ExLLM.chat(:anthropic, messages)
else
  IO.puts("Anthropic not configured")
end

context_stats(messages)

@spec context_stats(messages()) :: map()

Get statistics about message context usage.

Parameters

  • messages - List of conversation messages

Returns

Map with context statistics.

Examples

stats = ExLLM.context_stats(messages)
# => %{total_tokens: 1500, message_count: 10, ...}

context_window_size(provider, model)

@spec context_window_size(provider(), String.t()) :: non_neg_integer() | nil

Get default model for a provider.

Parameters

  • provider - LLM provider name
  • model - Model name

Returns

Context window size in tokens or nil if unknown.

Examples

tokens = ExLLM.context_window_size(:anthropic, "claude-3-5-sonnet-20241022")
# => 200000

cosine_similarity(embedding1, embedding2)

@spec cosine_similarity([float()], [float()]) :: float()

Calculate similarity between two embeddings.

Uses cosine similarity: 1.0 = identical, 0.0 = orthogonal, -1.0 = opposite

Examples

similarity = ExLLM.cosine_similarity(embedding1, embedding2)
# => 0.87

default_model(provider)

@spec default_model(provider()) :: String.t() | {:error, term()}

Get the default model for the specified provider.

Parameters

  • provider - The LLM provider

Returns

String model identifier.

Examples

model = ExLLM.default_model(:anthropic)
# => "claude-sonnet-4-20250514"

embeddings(provider, inputs, options \\ [])

@spec embeddings(provider(), [String.t()], options()) ::
  {:ok, ExLLM.Types.EmbeddingResponse.t()} | {:error, term()}

Generate embeddings for text inputs.

Embeddings are numerical representations of text that can be used for:

  • Semantic search
  • Clustering
  • Recommendations
  • Anomaly detection
  • Classification

Parameters

  • provider - The LLM provider (:openai, :anthropic, etc.)
  • inputs - List of text strings to embed
  • options - Options including :model, :dimensions, etc.

Options

  • :model - Embedding model to use (provider-specific)
  • :dimensions - Desired embedding dimensions (if supported)
  • :cache - Enable caching for embeddings
  • :cache_ttl - Cache TTL in milliseconds
  • :track_cost - Track embedding costs (default: true)

Examples

# Single embedding
{:ok, response} = ExLLM.embeddings(:openai, ["Hello, world!"])
[embedding] = response.embeddings
# => [0.0123, -0.0456, 0.0789, ...]

# Multiple embeddings
texts = ["First text", "Second text", "Third text"]
{:ok, response} = ExLLM.embeddings(:openai, texts,
  model: "text-embedding-3-small",
  dimensions: 256  # Reduce dimensions for storage
)

# With caching
{:ok, response} = ExLLM.embeddings(:openai, texts,
  cache: true,
  cache_ttl: :timer.hours(24)
)

estimate_tokens(text)

@spec estimate_tokens(String.t() | map() | [map()]) :: non_neg_integer()

Estimate token count for text.

Parameters

  • text - Text to analyze (string, message map, or list)

Returns

Estimated token count.

Examples

tokens = ExLLM.estimate_tokens("Hello, world!")
# => 4

execute_function(function_call, available_functions)

Execute a function call with available functions.

Examples

functions = [
  %{
    name: "get_weather",
    description: "Get weather",
    parameters: %{...},
    handler: fn args -> get_weather_impl(args) end
  }
]

{:ok, result} = ExLLM.execute_function(function_call, functions)

extract_text_from_image(provider, image_path, options \\ [])

@spec extract_text_from_image(provider(), String.t(), keyword()) ::
  {:ok, String.t()} | {:error, term()}

Extract text from an image using vision capabilities.

This is a convenience function for OCR-like tasks.

Examples

{:ok, text} = ExLLM.extract_text_from_image(:anthropic, "document.png")
IO.puts(text)

# With options
{:ok, text} = ExLLM.extract_text_from_image(:openai, "handwriting.jpg",
  model: "gpt-4-turbo",
  prompt: "Extract all text, preserving formatting"
)

find_models_with_features(required_features)

@spec find_models_with_features([atom()]) :: [{provider(), String.t()}]

Find models that support specific features.

Examples

# Find models with vision and function calling
models = ExLLM.find_models_with_features([:vision, :function_calling])
# => [{:openai, "gpt-4-turbo"}, {:anthropic, "claude-3-opus-20240229"}, ...]

# Find models that support streaming and have large context
models = ExLLM.find_models_with_features([:streaming])
|> Enum.filter(fn {provider, model} ->
  {:ok, info} = ExLLM.get_model_info(provider, model)
  info.context_window >= 100_000
end)

find_providers_with_features(features)

@spec find_providers_with_features([atom()]) :: [provider()]

Find providers that support all specified features.

Examples

# Find providers with embeddings and streaming
providers = ExLLM.find_providers_with_features([:embeddings, :streaming])
# => [:openai, :ollama]

# Find providers with vision and function calling
providers = ExLLM.find_providers_with_features([:vision, :function_calling])
# => [:openai, :anthropic, :gemini]

find_similar(query_embedding, items, options \\ [])

@spec find_similar([float()], [map()], keyword()) :: [map()]

Find the most similar items from a list of embeddings.

Options

  • :top_k - Number of results to return (default: 5)
  • :threshold - Minimum similarity score (default: 0.0)

Examples

query_embedding = get_embedding("search query")

items = [
  %{id: 1, text: "First doc", embedding: [...]},
  %{id: 2, text: "Second doc", embedding: [...]},
  # ...
]

results = ExLLM.find_similar(query_embedding, items,
  top_k: 10,
  threshold: 0.7
)
# => [%{item: %{id: 2, ...}, similarity: 0.92}, ...]

format_cost(cost)

@spec format_cost(float()) :: String.t()

Format cost for display.

Parameters

  • cost - Cost in dollars

Returns

Formatted cost string.

Examples

ExLLM.format_cost(0.0035)
# => "$0.003500"

format_function_result(result, provider)

@spec format_function_result(ExLLM.FunctionCalling.FunctionResult.t(), provider()) ::
  map()

Format function result for conversation continuation.

Examples

result = %FunctionCalling.FunctionResult{
  name: "get_weather",
  result: %{temperature: 72, condition: "sunny"}
}

formatted = ExLLM.format_function_result(result, :openai)

# Continue conversation with result
messages = messages ++ [formatted]
{:ok, response} = ExLLM.chat(:openai, messages)

get_model_info(provider, model_id)

@spec get_model_info(provider(), String.t()) ::
  {:ok, ExLLM.ModelCapabilities.ModelInfo.t()} | {:error, :not_found}

Get complete capability information for a model.

Examples

{:ok, info} = ExLLM.get_model_info(:openai, "gpt-4-turbo")

# Check capabilities
info.capabilities[:vision].supported
# => true

# Get context window
info.context_window
# => 128000

get_provider_capabilities(provider)

@spec get_provider_capabilities(provider()) ::
  {:ok, ExLLM.ProviderCapabilities.ProviderInfo.t()} | {:error, :not_found}

Get provider-level capabilities.

Provider capabilities are API-level features that are independent of specific models. This includes available endpoints, authentication methods, and provider limitations.

Examples

{:ok, caps} = ExLLM.get_provider_capabilities(:openai)
caps.endpoints
# => [:chat, :embeddings, :images, :audio, :completions, :fine_tuning, :files]

caps.features
# => [:streaming, :function_calling, :cost_tracking, :usage_tracking, ...]

get_session_messages(session, limit \\ nil)

@spec get_session_messages(ExLLM.Session.Types.Session.t(), non_neg_integer() | nil) ::
  [
    ExLLM.Session.Types.message()
  ]

Get messages from a session.

Parameters

  • session - The session to query
  • limit - Optional message limit

Returns

List of messages.

Examples

messages = ExLLM.get_session_messages(session)
last_10 = ExLLM.get_session_messages(session, 10)

is_local_provider?(provider)

@spec is_local_provider?(provider()) :: boolean()

Check if a provider is local (no API calls).

Examples

ExLLM.is_local_provider?(:ollama)
# => true

ExLLM.is_local_provider?(:openai)
# => false

list_embedding_models(provider, options \\ [])

@spec list_embedding_models(provider(), options()) ::
  {:ok, [ExLLM.Types.EmbeddingModel.t()]} | {:error, term()}

List available embedding models for a provider.

Examples

{:ok, models} = ExLLM.list_embedding_models(:openai)
Enum.each(models, fn m ->
  IO.puts("#{m.name} - #{m.dimensions} dimensions")
end)

list_model_features()

@spec list_model_features() :: [atom()]

List all trackable model features.

Examples

features = ExLLM.list_model_features()
# => [:streaming, :function_calling, :vision, :audio, ...]

list_models(provider, options \\ [])

@spec list_models(provider(), options()) ::
  {:ok, [ExLLM.Types.Model.t()]} | {:error, term()}

List available models for the specified provider.

Parameters

  • provider - The LLM provider
  • options - Options including configuration provider

Returns

{:ok, [%ExLLM.Types.Model{}]} on success, {:error, reason} on failure.

Examples

{:ok, models} = ExLLM.list_models(:anthropic)
Enum.each(models, fn model ->
  IO.puts(model.name)
end)

list_providers()

@spec list_providers() :: [provider()]

List all available providers.

Examples

providers = ExLLM.list_providers()
# => [:anthropic, :bedrock, :bumblebee, :gemini, :groq, :lmstudio, :mistral, :mock, :ollama, :openai, :openrouter, :perplexity, :xai]

list_recoverable_streams()

@spec list_recoverable_streams() :: [map()]

List all recoverable streams.

load_image(file_path, options \\ [])

@spec load_image(
  String.t(),
  keyword()
) :: {:ok, map()} | {:error, term()}

Load an image from file for use in vision requests.

Examples

{:ok, image_part} = ExLLM.load_image("photo.jpg")

message = %{
  role: "user",
  content: [
    ExLLM.Vision.text("What's in this image?"),
    image_part
  ]
}

load_session(json)

@spec load_session(String.t()) ::
  {:ok, ExLLM.Session.Types.Session.t()} | {:error, term()}

Load a session from JSON.

Parameters

  • json - JSON string containing session data

Returns

{:ok, session} on success, {:error, reason} on failure.

Examples

json = File.read!("session.json")
{:ok, session} = ExLLM.load_session(json)

model_supports?(provider, model_id, feature)

@spec model_supports?(provider(), String.t(), atom()) :: boolean()

Check if a model supports a specific feature.

Examples

ExLLM.model_supports?(:anthropic, "claude-3-opus-20240229", :vision)
# => true

ExLLM.model_supports?(:openai, "gpt-3.5-turbo", :vision)
# => false

models_by_capability(feature)

@spec models_by_capability(atom()) :: %{supported: list(), not_supported: list()}

Get models grouped by a specific capability.

Examples

vision_models = ExLLM.models_by_capability(:vision)
# => %{
#   supported: [{:openai, "gpt-4-turbo"}, {:anthropic, "claude-3-opus-20240229"}, ...],
#   not_supported: [{:openai, "gpt-3.5-turbo"}, ...]
# }

new_session(provider, opts \\ [])

@spec new_session(
  provider(),
  keyword()
) :: ExLLM.Session.Types.Session.t()

Create a new conversation session.

Parameters

  • provider - LLM provider to use for the session
  • opts - Session options (:name for session name)

Returns

A new session struct.

Examples

session = ExLLM.new_session(:anthropic)
session = ExLLM.new_session(:openai, name: "Customer Support")

parse_function_calls(response, provider)

@spec parse_function_calls(ExLLM.Types.LLMResponse.t() | map(), provider()) ::
  {:ok, [ExLLM.FunctionCalling.FunctionCall.t()]} | {:error, term()}

Parse function calls from an LLM response.

Examples

case ExLLM.parse_function_calls(response, :openai) do
  {:ok, [function_call | _]} ->
    # Execute the function
    execute_function(function_call)

  {:ok, []} ->
    # No function calls in response
    response.content
end

prepare_messages(messages, options \\ [])

@spec prepare_messages(messages(), options()) :: messages()

Prepare messages for sending to a provider with context management.

Parameters

  • messages - List of conversation messages
  • options - Options for context management

Options

  • :max_tokens - Maximum tokens for context (default: model-specific)
  • :strategy - Context truncation strategy (default: :sliding_window)
  • :preserve_messages - Number of recent messages to preserve (default: 5)

Returns

Prepared messages list that fits within context window.

Examples

messages = ExLLM.prepare_messages(long_conversation,
  max_tokens: 4000,
  strategy: :smart
)

provider_requires_auth?(provider)

@spec provider_requires_auth?(provider()) :: boolean()

Check if a provider requires authentication.

Examples

ExLLM.provider_requires_auth?(:openai)
# => true

ExLLM.provider_requires_auth?(:bumblebee)
# => false

provider_supports?(provider, feature)

@spec provider_supports?(provider(), atom()) :: boolean()

Check if a provider supports a specific feature or endpoint.

Examples

ExLLM.provider_supports?(:openai, :embeddings)
# => true

ExLLM.provider_supports?(:ollama, :cost_tracking)
# => false

ExLLM.provider_supports?(:anthropic, :computer_use)
# => true

recommend_models(requirements \\ [])

@spec recommend_models(keyword()) :: [{provider(), String.t(), map()}]

Get recommended models based on requirements.

Options

  • :features - Required features (list of atoms)
  • :min_context_window - Minimum context window size
  • :max_cost_per_1k_tokens - Maximum acceptable cost
  • :prefer_local - Prefer local models
  • :limit - Number of recommendations (default: 5)

Examples

# Find best models for vision tasks with large context
recommendations = ExLLM.recommend_models(
  features: [:vision, :streaming],
  min_context_window: 50_000,
  prefer_local: false
)

# Find cheapest models for basic chat
recommendations = ExLLM.recommend_models(
  features: [:multi_turn, :system_messages],
  max_cost_per_1k_tokens: 1.0
)

recommend_providers(requirements \\ %{})

@spec recommend_providers(map()) :: [map()]

Get provider recommendations based on requirements.

Parameters

  • requirements - Map with:
    • :required_features - Features that must be supported
    • :preferred_features - Nice-to-have features
    • :required_endpoints - Required API endpoints
    • :exclude_providers - Providers to exclude
    • :prefer_local - Prefer local providers (default: false)
    • :prefer_free - Prefer free providers (default: false)

Examples

# Find best providers for multimodal AI
recommendations = ExLLM.recommend_providers(%{
  required_features: [:vision, :streaming],
  preferred_features: [:audio_input, :function_calling],
  exclude_providers: [:mock]
})
# => [
#   %{provider: :openai, score: 0.95, matched_features: [...], missing_features: []},
#   %{provider: :anthropic, score: 0.80, matched_features: [...], missing_features: [...]}
# ]

# Find free local providers
recommendations = ExLLM.recommend_providers(%{
  required_features: [:chat],
  prefer_local: true,
  prefer_free: true
})

resume_stream(recovery_id, opts \\ [])

@spec resume_stream(
  String.t(),
  keyword()
) :: {:ok, ExLLM.Types.stream()} | {:error, term()}

Resume a previously interrupted stream.

Options

  • :strategy - Recovery strategy (:exact, :paragraph, :summarize)

Examples

{:ok, resumed_stream} = ExLLM.resume_stream(recovery_id)

save_session(session)

@spec save_session(ExLLM.Session.Types.Session.t()) ::
  {:ok, String.t()} | {:error, term()}

Save a session to JSON.

Parameters

  • session - The session to save

Returns

{:ok, json} on success, {:error, reason} on failure.

Examples

{:ok, json} = ExLLM.save_session(session)
File.write!("session.json", json)

session_token_usage(session)

@spec session_token_usage(ExLLM.Session.Types.Session.t()) :: non_neg_integer()

Get total token usage for a session.

Parameters

  • session - The session to analyze

Returns

Total token count.

Examples

tokens = ExLLM.session_token_usage(session)
# => 2500

stream_chat(provider_or_model, messages, options \\ [])

@spec stream_chat(provider() | String.t(), messages(), options()) ::
  {:ok, ExLLM.Types.stream()} | {:error, term()}

Send a streaming chat completion request to the specified LLM provider.

Parameters

  • provider - The LLM provider (:anthropic, :openai, :ollama)
  • messages - List of conversation messages
  • options - Options for the request (see module docs)

Options

Same as chat/3, plus:

  • :on_chunk - Callback function for each chunk
  • :stream_recovery - Enable automatic stream recovery (default: false)
  • :recovery_strategy - How to resume: :exact, :paragraph, or :summarize
  • :recovery_id - Custom ID for recovery (auto-generated if not provided)

Returns

{:ok, stream} on success where stream yields %ExLLM.Types.StreamChunk{} structs, {:error, reason} on failure.

Examples

{:ok, stream} = ExLLM.stream_chat(:anthropic, messages)

# Process the stream
for chunk <- stream do
  case chunk do
    %{content: content} when content != nil ->
      IO.write(content)
    %{finish_reason: "stop"} ->
      IO.puts("\nDone!")
    _ ->
      :continue
  end
end

# With context management
{:ok, stream} = ExLLM.stream_chat(:anthropic, messages,
  max_tokens: 4000,
  strategy: :smart
)

supported_providers()

@spec supported_providers() :: [provider()]

Get list of supported providers.

Returns

List of provider atoms.

Examples

providers = ExLLM.supported_providers()
# => [:anthropic, :openai, :ollama]

supports_vision?(provider, model)

@spec supports_vision?(provider(), String.t()) :: boolean()

Check if a provider and model support vision inputs.

Examples

ExLLM.supports_vision?(:anthropic, "claude-3-opus-20240229")
# => true

ExLLM.supports_vision?(:openai, "gpt-3.5-turbo")
# => false

validate_context(messages, options \\ [])

@spec validate_context(messages(), options()) ::
  {:ok, non_neg_integer()} | {:error, String.t()}

Validate that messages fit within a model's context window.

Parameters

  • messages - List of conversation messages
  • options - Options including model info

Returns

{:ok, token_count} if valid, {:error, reason} if too large.

vision_message(text, image_sources, options \\ [])

@spec vision_message(String.t(), [String.t()], keyword()) ::
  {:ok, ExLLM.Types.message()} | {:error, term()}

Create a vision-enabled message with text and images.

Examples

# With image URLs
{:ok, message} = ExLLM.vision_message("What's in these images?", [
  "https://example.com/image1.jpg",
  "https://example.com/image2.png"
])

{:ok, response} = ExLLM.chat(:anthropic, [message])

# With local images
{:ok, message} = ExLLM.vision_message("Describe these photos", [
  "/path/to/photo1.jpg",
  "/path/to/photo2.png"
])

# With options
{:ok, message} = ExLLM.vision_message("Analyze this chart", 
  ["chart.png"],
  role: "user",
  detail: :high  # High detail for complex images
)