ExLLM (ex_llm v0.8.1)
View SourceExLLM - Unified Elixir client library for Large Language Models.
ExLLM provides a consistent interface across multiple LLM providers including OpenAI, Anthropic Claude, Ollama, and others. It features configuration injection, standardized error handling, and streaming support.
Quick Start
# Using environment variables
messages = [%{role: "user", content: "Hello!"}]
{:ok, response} = ExLLM.chat(:anthropic, messages)
IO.puts(response.content)
# Using static configuration
config = %{anthropic: %{api_key: "your-key"}}
{:ok, provider} = ExLLM.Infrastructure.ConfigProvider.Static.start_link(config)
{:ok, response} = ExLLM.chat(:anthropic, messages, config_provider: provider)
Supported Providers
:anthropic
- Anthropic Claude models:openai
- OpenAI GPT models:groq
- Groq (fast inference):lmstudio
- Local models via LM Studio (desktop app):mistral
- Mistral AI models (European AI):perplexity
- Perplexity AI models (search-augmented):openrouter
- OpenRouter (300+ models from multiple providers):ollama
- Local models via Ollama:bedrock
- AWS Bedrock (multiple providers):gemini
- Google Gemini models:xai
- X.AI Grok models:bumblebee
- Local models via Bumblebee (Phi-2, Llama 2, Mistral, etc.):mock
- Mock provider for testing
Features
- Unified Interface: Same API across all providers
- Configuration Injection: Flexible config management
- Streaming Support: Real-time response streaming with error recovery
- Error Standardization: Consistent error handling
- Function Calling: Unified interface for tool use across providers
- Model Discovery: Query and compare model capabilities
- Automatic Retries: Exponential backoff with provider-specific policies
- Mock Adapter: Built-in testing support without API calls
- Cost Tracking: Automatic usage and cost calculation
- Context Management: Automatic message truncation for model limits
- Session Management: Conversation state tracking
- Structured Outputs: Schema validation via instructor integration
- No Process Dependencies: Pure functional core
- Extensible: Easy to add new providers
Configuration
ExLLM supports multiple configuration methods:
Environment Variables
export ANTHROPIC_API_KEY="api-..."
export OPENAI_API_KEY="sk-..."
export GROQ_API_KEY="gsk-..."
export OPENROUTER_API_KEY="sk-or-..."
export OLLAMA_API_BASE="http://localhost:11434"
export GOOGLE_API_KEY="your-key"
export XAI_API_KEY="xai-..."
export AWS_ACCESS_KEY_ID="your-key"
export AWS_SECRET_ACCESS_KEY="your-secret"
Static Configuration
config = %{
anthropic: %{api_key: "api-...", model: "claude-3-5-sonnet-20241022"},
openai: %{api_key: "sk-...", model: "gpt-4"},
openrouter: %{api_key: "sk-or-...", model: "openai/gpt-4o"},
ollama: %{base_url: "http://localhost:11434", model: "llama2"},
bedrock: %{access_key_id: "...", secret_access_key: "...", region: "us-east-1"},
gemini: %{api_key: "...", model: "gemini-pro"},
bumblebee: %{model: "microsoft/phi-2"}
}
{:ok, provider} = ExLLM.Infrastructure.ConfigProvider.Static.start_link(config)
Custom Configuration
defmodule MyConfigProvider do
@behaviour ExLLM.Infrastructure.ConfigProvider
def get([:anthropic, :api_key]), do: MyApp.get_secret("anthropic_key")
def get(_), do: nil
def get_all(), do: %{}
end
Examples
# Simple chat
{:ok, response} = ExLLM.chat(:anthropic, [
%{role: "user", content: "What is Elixir?"}
])
# With options
{:ok, response} = ExLLM.chat(:anthropic, messages,
model: "claude-3-haiku-20240307",
temperature: 0.7,
max_tokens: 1000
)
# Streaming
{:ok, stream} = ExLLM.stream_chat(:anthropic, messages)
for chunk <- stream do
if chunk.content, do: IO.write(chunk.content)
end
# Check if provider is configured
if ExLLM.configured?(:anthropic) do
{:ok, response} = ExLLM.chat(:anthropic, messages)
end
# List available models
{:ok, models} = ExLLM.list_models(:anthropic)
Enum.each(models, fn model ->
IO.puts(model.name)
end)
Summary
Functions
Add a message to a session.
Analyze images with a specific prompt.
Calculate cost for token usage.
Send a chat completion request to the specified LLM provider.
Send a chat request using a session, automatically tracking messages and usage.
Clear messages from a session while preserving metadata.
Compare capabilities across multiple models.
Compare capabilities across multiple providers.
Check if the specified provider is properly configured.
Get statistics about message context usage.
Get default model for a provider.
Calculate similarity between two embeddings.
Get the default model for the specified provider.
Generate embeddings for text inputs.
Estimate token count for text.
Execute a function call with available functions.
Extract text from an image using vision capabilities.
Find models that support specific features.
Find providers that support all specified features.
Find the most similar items from a list of embeddings.
Format cost for display.
Format function result for conversation continuation.
Get complete capability information for a model.
Get provider-level capabilities.
Get messages from a session.
Check if a provider is local (no API calls).
List available embedding models for a provider.
List all trackable model features.
List available models for the specified provider.
List all available providers.
List all recoverable streams.
Load an image from file for use in vision requests.
Load a session from JSON.
Check if a model supports a specific feature.
Get models grouped by a specific capability.
Create a new conversation session.
Parse function calls from an LLM response.
Prepare messages for sending to a provider with context management.
Check if a provider requires authentication.
Check if a provider supports a specific feature or endpoint.
Get recommended models based on requirements.
Get provider recommendations based on requirements.
Resume a previously interrupted stream.
Save a session to JSON.
Get total token usage for a session.
Send a streaming chat completion request to the specified LLM provider.
Get list of supported providers.
Check if a provider and model support vision inputs.
Validate that messages fit within a model's context window.
Create a vision-enabled message with text and images.
Types
@type messages() :: [ExLLM.Types.message()]
@type options() :: keyword()
@type provider() ::
:anthropic
| :bumblebee
| :openai
| :groq
| :lmstudio
| :mistral
| :perplexity
| :openrouter
| :ollama
| :bedrock
| :gemini
| :xai
| :mock
Functions
@spec add_session_message(ExLLM.Types.Session.t(), String.t(), String.t(), keyword()) :: ExLLM.Types.Session.t()
Add a message to a session.
Parameters
session
- The session to updaterole
- Message role ("user", "assistant", etc.)content
- Message contentopts
- Additional message metadata
Returns
Updated session.
Examples
session = ExLLM.add_session_message(session, "user", "What is Elixir?")
@spec analyze_images(provider(), [String.t()], String.t(), keyword()) :: {:ok, String.t()} | {:error, term()}
Analyze images with a specific prompt.
Examples
{:ok, analysis} = ExLLM.analyze_images(:anthropic,
["chart1.png", "chart2.png"],
"Compare these two charts and identify key differences"
)
@spec calculate_cost(provider(), String.t(), ExLLM.Types.token_usage()) :: ExLLM.Types.cost_result() | %{error: String.t()}
Calculate cost for token usage.
Parameters
provider
- LLM provider namemodel
- Model nametoken_usage
- Map with:input_tokens
and:output_tokens
Returns
Cost calculation result or error map.
Examples
usage = %{input_tokens: 1000, output_tokens: 500}
cost = ExLLM.calculate_cost("openai", "gpt-4", usage)
# => %{total_cost: 0.06, ...}
@spec chat(provider() | String.t(), messages(), options()) :: {:ok, ExLLM.Types.LLMResponse.t() | struct() | map()} | {:error, term()}
Send a chat completion request to the specified LLM provider.
Parameters
provider
- The LLM provider (:anthropic
,:openai
,:groq
, etc.) or a model string like "groq/llama3-70b"messages
- List of conversation messagesoptions
- Options for the request (see module docs)
Options
:model
- Override the default model:temperature
- Temperature setting (0.0 to 1.0):max_tokens
- Maximum tokens in response or context:config_provider
- Configuration provider module or pid:track_cost
- Whether to track costs (default: true):strategy
- Context truncation strategy (default: :sliding_window):sliding_window
- Keep most recent messages:smart
- Preserve system messages and recent context
:preserve_messages
- Number of recent messages to always preserve (default: 5):response_model
- Ecto schema or type spec for structured output (requires instructor):max_retries
- Number of retries for structured output validation:functions
- List of available functions for function calling:function_call
- Control function calling: "auto", "none", or specific function:tools
- Alternative to functions for providers that use tools API:retry
- Enable automatic retry (default: true):retry_count
- Number of retry attempts (default: 3):retry_delay
- Initial retry delay in ms (default: 1000):retry_backoff
- Backoff strategy: :exponential or :linear (default: :exponential):retry_jitter
- Add jitter to retry delays (default: true):stream_recovery
- Enable stream recovery (default: false):recovery_strategy
- Recovery strategy: :exact, :paragraph, or :summarize:cache
- Enable caching for this request (default: false unless globally enabled):cache_ttl
- Cache TTL in milliseconds (default: 15 minutes):timeout
- Request timeout in milliseconds (provider-specific defaults)- Ollama default: 120000 (2 minutes)
- Other providers use their client library defaults
- Mock provider options:
:mock_response
- Static response or response map:mock_handler
- Function to generate dynamic responses:mock_error
- Simulate an error:mock_chunks
- List of chunks for streaming:chunk_delay
- Delay between chunks in ms:capture_requests
- Capture requests for testing
Returns
{:ok, %ExLLM.Types.LLMResponse{}}
on success, or {:ok, struct}
when using
response_model. Returns {:error, reason}
on failure.
Examples
# Simple usage
{:ok, response} = ExLLM.chat(:anthropic, [
%{role: "user", content: "Hello!"}
])
# Using provider/model syntax
{:ok, response} = ExLLM.chat("groq/llama3-70b", [
%{role: "user", content: "Hello!"}
])
# Groq with specific model
{:ok, response} = ExLLM.chat(:groq, messages, model: "mixtral-8x7b-32768")
# With custom configuration
{:ok, provider} = ExLLM.Infrastructure.ConfigProvider.Static.start_link(%{
anthropic: %{api_key: "your-key"}
})
{:ok, response} = ExLLM.chat(:anthropic, messages, config_provider: provider)
# With model override
{:ok, response} = ExLLM.chat(:openai, messages, model: "gpt-4-turbo")
# With context management
{:ok, response} = ExLLM.chat(:anthropic, messages,
max_tokens: 4000,
strategy: :smart
)
# With structured output (requires instructor)
{:ok, classification} = ExLLM.chat(:anthropic, messages,
response_model: EmailClassification,
max_retries: 3
)
# With function calling
functions = [
%{
name: "get_weather",
description: "Get current weather",
parameters: %{
type: "object",
properties: %{location: %{type: "string"}},
required: ["location"]
}
}
]
{:ok, response} = ExLLM.chat(:openai, messages,
functions: functions,
function_call: "auto"
)
@spec chat_with_session(ExLLM.Types.Session.t(), String.t(), options()) :: {:ok, {ExLLM.Types.LLMResponse.t(), ExLLM.Types.Session.t()}} | {:error, term()}
Send a chat request using a session, automatically tracking messages and usage.
Parameters
session
- The session to usecontent
- The user message contentoptions
- Chat options (same aschat/3
)
Returns
{:ok, {response, updated_session}}
on success, {:error, reason}
on failure.
Examples
session = ExLLM.new_session(:anthropic)
{:ok, {response, session}} = ExLLM.chat_with_session(session, "Hello!")
# Session now contains the conversation history
@spec clear_session(ExLLM.Types.Session.t()) :: ExLLM.Types.Session.t()
Clear messages from a session while preserving metadata.
Parameters
session
- The session to clear
Returns
Updated session with no messages.
Examples
session = ExLLM.clear_session(session)
Compare capabilities across multiple models.
Examples
comparison = ExLLM.compare_models([
{:openai, "gpt-4-turbo"},
{:anthropic, "claude-3-5-sonnet-20241022"},
{:gemini, "gemini-pro"}
])
# See which features each model supports
comparison.features[:vision]
# => [%{supported: true}, %{supported: true}, %{supported: false}]
Compare capabilities across multiple providers.
Examples
comparison = ExLLM.compare_providers([:openai, :anthropic, :ollama])
# See all features across providers
comparison.features
# => [:streaming, :function_calling, :vision, ...]
# Check specific provider capabilities
comparison.comparison.openai.features
# => [:streaming, :function_calling, :cost_tracking, ...]
Check if the specified provider is properly configured.
Parameters
provider
- The LLM provider to checkoptions
- Options including configuration provider
Returns
true
if configured, false
otherwise.
Examples
if ExLLM.configured?(:anthropic) do
{:ok, response} = ExLLM.chat(:anthropic, messages)
else
IO.puts("Anthropic not configured")
end
Get statistics about message context usage.
Parameters
messages
- List of conversation messages
Returns
Map with context statistics.
Examples
stats = ExLLM.context_stats(messages)
# => %{total_tokens: 1500, message_count: 10, ...}
@spec context_window_size(provider(), String.t()) :: non_neg_integer() | nil
Get default model for a provider.
Parameters
provider
- LLM provider namemodel
- Model name
Returns
Context window size in tokens or nil if unknown.
Examples
tokens = ExLLM.context_window_size(:anthropic, "claude-3-5-sonnet-20241022")
# => 200000
Calculate similarity between two embeddings.
Uses cosine similarity: 1.0 = identical, 0.0 = orthogonal, -1.0 = opposite
Examples
similarity = ExLLM.cosine_similarity(embedding1, embedding2)
# => 0.87
Get the default model for the specified provider.
Parameters
provider
- The LLM provider
Returns
String model identifier.
Examples
model = ExLLM.default_model(:anthropic)
# => "claude-sonnet-4-20250514"
@spec embeddings(provider(), [String.t()], options()) :: {:ok, ExLLM.Types.EmbeddingResponse.t()} | {:error, term()}
Generate embeddings for text inputs.
Embeddings are numerical representations of text that can be used for:
- Semantic search
- Clustering
- Recommendations
- Anomaly detection
- Classification
Parameters
provider
- The LLM provider (:openai
,:anthropic
, etc.)inputs
- List of text strings to embedoptions
- Options including:model
,:dimensions
, etc.
Options
:model
- Embedding model to use (provider-specific):dimensions
- Desired embedding dimensions (if supported):cache
- Enable caching for embeddings:cache_ttl
- Cache TTL in milliseconds:track_cost
- Track embedding costs (default: true)
Examples
# Single embedding
{:ok, response} = ExLLM.embeddings(:openai, ["Hello, world!"])
[embedding] = response.embeddings
# => [0.0123, -0.0456, 0.0789, ...]
# Multiple embeddings
texts = ["First text", "Second text", "Third text"]
{:ok, response} = ExLLM.embeddings(:openai, texts,
model: "text-embedding-3-small",
dimensions: 256 # Reduce dimensions for storage
)
# With caching
{:ok, response} = ExLLM.embeddings(:openai, texts,
cache: true,
cache_ttl: :timer.hours(24)
)
@spec estimate_tokens(String.t() | map() | [map()]) :: non_neg_integer()
Estimate token count for text.
Parameters
text
- Text to analyze (string, message map, or list)
Returns
Estimated token count.
Examples
tokens = ExLLM.estimate_tokens("Hello, world!")
# => 4
@spec execute_function(ExLLM.Core.FunctionCalling.FunctionCall.t(), [map()]) :: {:ok, ExLLM.Core.FunctionCalling.FunctionResult.t()} | {:error, ExLLM.Core.FunctionCalling.FunctionResult.t()}
Execute a function call with available functions.
Examples
functions = [
%{
name: "get_weather",
description: "Get weather",
parameters: %{...},
handler: fn args -> get_weather_impl(args) end
}
]
{:ok, result} = ExLLM.execute_function(function_call, functions)
@spec extract_text_from_image(provider(), String.t(), keyword()) :: {:ok, String.t()} | {:error, term()}
Extract text from an image using vision capabilities.
This is a convenience function for OCR-like tasks.
Examples
{:ok, text} = ExLLM.extract_text_from_image(:anthropic, "document.png")
IO.puts(text)
# With options
{:ok, text} = ExLLM.extract_text_from_image(:openai, "handwriting.jpg",
model: "gpt-4-turbo",
prompt: "Extract all text, preserving formatting"
)
Find models that support specific features.
Examples
# Find models with vision and function calling
models = ExLLM.find_models_with_features([:vision, :function_calling])
# => [{:openai, "gpt-4-turbo"}, {:anthropic, "claude-3-opus-20240229"}, ...]
# Find models that support streaming and have large context
models = ExLLM.find_models_with_features([:streaming])
|> Enum.filter(fn {provider, model} ->
{:ok, info} = ExLLM.get_model_info(provider, model)
info.context_window >= 100_000
end)
Find providers that support all specified features.
Examples
# Find providers with embeddings and streaming
providers = ExLLM.find_providers_with_features([:embeddings, :streaming])
# => [:openai, :ollama]
# Find providers with vision and function calling
providers = ExLLM.find_providers_with_features([:vision, :function_calling])
# => [:openai, :anthropic, :gemini]
Find the most similar items from a list of embeddings.
Options
:top_k
- Number of results to return (default: 5):threshold
- Minimum similarity score (default: 0.0)
Examples
query_embedding = get_embedding("search query")
items = [
%{id: 1, text: "First doc", embedding: [...]},
%{id: 2, text: "Second doc", embedding: [...]},
# ...
]
results = ExLLM.find_similar(query_embedding, items,
top_k: 10,
threshold: 0.7
)
# => [%{item: %{id: 2, ...}, similarity: 0.92}, ...]
Format cost for display.
Parameters
cost
- Cost in dollars
Returns
Formatted cost string.
Examples
ExLLM.format_cost(0.0035)
# => "$0.003500"
@spec format_function_result( ExLLM.Core.FunctionCalling.FunctionResult.t(), provider() ) :: map()
Format function result for conversation continuation.
Examples
result = %FunctionCalling.FunctionResult{
name: "get_weather",
result: %{temperature: 72, condition: "sunny"}
}
formatted = ExLLM.format_function_result(result, :openai)
# Continue conversation with result
messages = messages ++ [formatted]
{:ok, response} = ExLLM.chat(:openai, messages)
@spec get_model_info(provider(), String.t()) :: {:ok, ExLLM.Infrastructure.Config.ModelCapabilities.ModelInfo.t()} | {:error, :not_found}
Get complete capability information for a model.
Examples
{:ok, info} = ExLLM.get_model_info(:openai, "gpt-4-turbo")
# Check capabilities
info.capabilities[:vision].supported
# => true
# Get context window
info.context_window
# => 128000
@spec get_provider_capabilities(provider()) :: {:ok, ExLLM.Infrastructure.Config.ProviderCapabilities.ProviderInfo.t()} | {:error, :not_found}
Get provider-level capabilities.
Provider capabilities are API-level features that are independent of specific models. This includes available endpoints, authentication methods, and provider limitations.
Examples
{:ok, caps} = ExLLM.get_provider_capabilities(:openai)
caps.endpoints
# => [:chat, :embeddings, :images, :audio, :completions, :fine_tuning, :files]
caps.features
# => [:streaming, :function_calling, :cost_tracking, :usage_tracking, ...]
@spec get_session_messages(ExLLM.Types.Session.t(), non_neg_integer() | nil) :: [ ExLLM.Types.message() ]
Get messages from a session.
Parameters
session
- The session to querylimit
- Optional message limit
Returns
List of messages.
Examples
messages = ExLLM.get_session_messages(session)
last_10 = ExLLM.get_session_messages(session, 10)
Check if a provider is local (no API calls).
Examples
ExLLM.is_local_provider?(:ollama)
# => true
ExLLM.is_local_provider?(:openai)
# => false
@spec list_embedding_models(provider(), options()) :: {:ok, [ExLLM.Types.EmbeddingModel.t()]} | {:error, term()}
List available embedding models for a provider.
Examples
{:ok, models} = ExLLM.list_embedding_models(:openai)
Enum.each(models, fn m ->
IO.puts("#{m.name} - #{m.dimensions} dimensions")
end)
@spec list_model_features() :: [atom()]
List all trackable model features.
Examples
features = ExLLM.list_model_features()
# => [:streaming, :function_calling, :vision, :audio, ...]
@spec list_models(provider(), options()) :: {:ok, [ExLLM.Types.Model.t()]} | {:error, term()}
List available models for the specified provider.
Parameters
provider
- The LLM provideroptions
- Options including configuration provider
Returns
{:ok, [%ExLLM.Types.Model{}]}
on success, {:error, reason}
on failure.
Examples
{:ok, models} = ExLLM.list_models(:anthropic)
Enum.each(models, fn model ->
IO.puts(model.name)
end)
@spec list_providers() :: [provider()]
List all available providers.
Examples
providers = ExLLM.list_providers()
# => [:anthropic, :bedrock, :bumblebee, :gemini, :groq, :lmstudio, :mistral, :mock, :ollama, :openai, :openrouter, :perplexity, :xai]
@spec list_recoverable_streams() :: [map()]
List all recoverable streams.
Load an image from file for use in vision requests.
Examples
{:ok, image_part} = ExLLM.load_image("photo.jpg")
message = %{
role: "user",
content: [
ExLLM.Core.Vision.text("What's in this image?"),
image_part
]
}
@spec load_session(String.t()) :: {:ok, ExLLM.Types.Session.t()} | {:error, term()}
Load a session from JSON.
Parameters
json
- JSON string containing session data
Returns
{:ok, session}
on success, {:error, reason}
on failure.
Examples
json = File.read!("session.json")
{:ok, session} = ExLLM.load_session(json)
Check if a model supports a specific feature.
Examples
ExLLM.model_supports?(:anthropic, "claude-3-opus-20240229", :vision)
# => true
ExLLM.model_supports?(:openai, "gpt-3.5-turbo", :vision)
# => false
Get models grouped by a specific capability.
Examples
vision_models = ExLLM.models_by_capability(:vision)
# => %{
# supported: [{:openai, "gpt-4-turbo"}, {:anthropic, "claude-3-opus-20240229"}, ...],
# not_supported: [{:openai, "gpt-3.5-turbo"}, ...]
# }
@spec new_session( provider(), keyword() ) :: ExLLM.Types.Session.t()
Create a new conversation session.
Parameters
provider
- LLM provider to use for the sessionopts
- Session options (:name
for session name)
Returns
A new session struct.
Examples
session = ExLLM.new_session(:anthropic)
session = ExLLM.new_session(:openai, name: "Customer Support")
@spec parse_function_calls(ExLLM.Types.LLMResponse.t() | map(), provider()) :: {:ok, [ExLLM.Core.FunctionCalling.FunctionCall.t()]} | {:error, term()}
Parse function calls from an LLM response.
Examples
case ExLLM.parse_function_calls(response, :openai) do
{:ok, [function_call | _]} ->
# Execute the function
execute_function(function_call)
{:ok, []} ->
# No function calls in response
response.content
end
Prepare messages for sending to a provider with context management.
Parameters
messages
- List of conversation messagesoptions
- Options for context management
Options
:max_tokens
- Maximum tokens for context (default: model-specific):strategy
- Context truncation strategy (default: :sliding_window):preserve_messages
- Number of recent messages to preserve (default: 5)
Returns
Prepared messages list that fits within context window.
Examples
messages = ExLLM.prepare_messages(long_conversation,
max_tokens: 4000,
strategy: :smart
)
Check if a provider requires authentication.
Examples
ExLLM.provider_requires_auth?(:openai)
# => true
ExLLM.provider_requires_auth?(:bumblebee)
# => false
Check if a provider supports a specific feature or endpoint.
Examples
ExLLM.provider_supports?(:openai, :embeddings)
# => true
ExLLM.provider_supports?(:ollama, :cost_tracking)
# => false
ExLLM.provider_supports?(:anthropic, :computer_use)
# => true
Get recommended models based on requirements.
Options
:features
- Required features (list of atoms):min_context_window
- Minimum context window size:max_cost_per_1k_tokens
- Maximum acceptable cost:prefer_local
- Prefer local models:limit
- Number of recommendations (default: 5)
Examples
# Find best models for vision tasks with large context
recommendations = ExLLM.recommend_models(
features: [:vision, :streaming],
min_context_window: 50_000,
prefer_local: false
)
# Find cheapest models for basic chat
recommendations = ExLLM.recommend_models(
features: [:multi_turn, :system_messages],
max_cost_per_1k_tokens: 1.0
)
Get provider recommendations based on requirements.
Parameters
requirements
- Map with::required_features
- Features that must be supported:preferred_features
- Nice-to-have features:required_endpoints
- Required API endpoints:exclude_providers
- Providers to exclude:prefer_local
- Prefer local providers (default: false):prefer_free
- Prefer free providers (default: false)
Examples
# Find best providers for multimodal AI
recommendations = ExLLM.recommend_providers(%{
required_features: [:vision, :streaming],
preferred_features: [:audio_input, :function_calling],
exclude_providers: [:mock]
})
# => [
# %{provider: :openai, score: 0.95, matched_features: [...], missing_features: []},
# %{provider: :anthropic, score: 0.80, matched_features: [...], missing_features: [...]}
# ]
# Find free local providers
recommendations = ExLLM.recommend_providers(%{
required_features: [:chat],
prefer_local: true,
prefer_free: true
})
@spec resume_stream( String.t(), keyword() ) :: {:ok, ExLLM.Types.stream()} | {:error, term()}
Resume a previously interrupted stream.
Options
:strategy
- Recovery strategy (:exact, :paragraph, :summarize)
Examples
{:ok, resumed_stream} = ExLLM.resume_stream(recovery_id)
@spec save_session(ExLLM.Types.Session.t()) :: {:ok, String.t()} | {:error, term()}
Save a session to JSON.
Parameters
session
- The session to save
Returns
{:ok, json}
on success, {:error, reason}
on failure.
Examples
{:ok, json} = ExLLM.save_session(session)
File.write!("session.json", json)
@spec session_token_usage(ExLLM.Types.Session.t()) :: non_neg_integer()
Get total token usage for a session.
Parameters
session
- The session to analyze
Returns
Total token count.
Examples
tokens = ExLLM.session_token_usage(session)
# => 2500
@spec stream_chat(provider() | String.t(), messages(), options()) :: {:ok, ExLLM.Types.stream()} | {:error, term()}
Send a streaming chat completion request to the specified LLM provider.
Parameters
provider
- The LLM provider (:anthropic
,:openai
,:ollama
)messages
- List of conversation messagesoptions
- Options for the request (see module docs)
Options
Same as chat/3
, plus:
:on_chunk
- Callback function for each chunk:stream_recovery
- Enable automatic stream recovery (default: false):recovery_strategy
- How to resume: :exact, :paragraph, or :summarize:recovery_id
- Custom ID for recovery (auto-generated if not provided)
Returns
{:ok, stream}
on success where stream yields %ExLLM.Types.StreamChunk{}
structs,
{:error, reason}
on failure.
Examples
{:ok, stream} = ExLLM.stream_chat(:anthropic, messages)
# Process the stream
for chunk <- stream do
case chunk do
%{content: content} when content != nil ->
IO.write(content)
%{finish_reason: "stop"} ->
IO.puts("\nDone!")
_ ->
:continue
end
end
# With context management
{:ok, stream} = ExLLM.stream_chat(:anthropic, messages,
max_tokens: 4000,
strategy: :smart
)
@spec supported_providers() :: [provider()]
Get list of supported providers.
Returns
List of provider atoms.
Examples
providers = ExLLM.supported_providers()
# => [:anthropic, :openai, :ollama]
Check if a provider and model support vision inputs.
Examples
ExLLM.supports_vision?(:anthropic, "claude-3-opus-20240229")
# => true
ExLLM.supports_vision?(:openai, "gpt-3.5-turbo")
# => false
@spec validate_context(messages(), options()) :: {:ok, non_neg_integer()} | {:error, String.t()}
Validate that messages fit within a model's context window.
Parameters
messages
- List of conversation messagesoptions
- Options including model info
Returns
{:ok, token_count}
if valid, {:error, reason}
if too large.
@spec vision_message(String.t(), [String.t()], keyword()) :: {:ok, ExLLM.Types.message()} | {:error, term()}
Create a vision-enabled message with text and images.
Examples
# With image URLs
{:ok, message} = ExLLM.vision_message("What's in these images?", [
"https://example.com/image1.jpg",
"https://example.com/image2.png"
])
{:ok, response} = ExLLM.chat(:anthropic, [message])
# With local images
{:ok, message} = ExLLM.vision_message("Describe these photos", [
"/path/to/photo1.jpg",
"/path/to/photo2.png"
])
# With options
{:ok, message} = ExLLM.vision_message("Analyze this chart",
["chart.png"],
role: "user",
detail: :high # High detail for complex images
)