Provider Capabilities Guide
View SourceThis document explains how to find and update provider capabilities in ExLLM.
Current Provider Capabilities
Provider capabilities are defined in lib/ex_llm/provider_capabilities.ex
. Each provider has:
- endpoints: API endpoints available (e.g.,
:chat
,:embeddings
,:images
) - features: Capabilities supported (e.g.,
:streaming
,:vision
,:function_calling
) - authentication: Auth methods supported (e.g.,
:api_key
,:oauth2
) - limitations: Known limitations and constraints
How to Find Provider Capabilities
1. Official Documentation
Always check the official documentation for the most accurate information:
OpenAI: https://platform.openai.com/docs/api-reference
- Models: https://platform.openai.com/docs/models
- Capabilities: Check individual endpoint docs
Anthropic: https://docs.anthropic.com/
Google Gemini: https://ai.google.dev/docs
- Models: https://ai.google.dev/models/gemini
- API capabilities: https://ai.google.dev/api/rest
AWS Bedrock: https://docs.aws.amazon.com/bedrock/
X.AI: https://docs.x.ai/
- API reference: https://docs.x.ai/api
2. API Discovery Endpoints
Some providers offer API endpoints to discover capabilities:
# OpenAI - List available models
{:ok, models} = ExLLM.list_models(:openai)
# Check if a model supports specific features
{:ok, model_info} = ExLLM.ModelCapabilities.get_capabilities(:openai, "gpt-4o")
3. Feature Mapping
Here's how common features map across providers:
Feature | OpenAI | Anthropic | Gemini | Bedrock |
---|---|---|---|---|
Chat | ✓ /v1/chat/completions | ✓ /v1/messages | ✓ /v1beta/models/.../generateContent | ✓ /invoke |
Embeddings | ✓ /v1/embeddings | ✗ | ✓ /v1beta/models/.../embedContent | ✓ (varies) |
Images | ✓ /v1/images/generations | ✗ | ✗ | ✗ (use Stable Diffusion) |
Vision | ✓ (GPT-4V) | ✓ (Claude 3) | ✓ (Gemini Pro Vision) | ✓ (varies) |
Function Calling | ✓ | ✓ (Tools) | ✓ | ✓ (varies) |
Streaming | ✓ | ✓ | ✓ | ✓ |
JSON Mode | ✓ | ✓ | ✓ | ✓ (varies) |
Updating Provider Capabilities
When updating capabilities in provider_capabilities.ex
:
- Check Official Docs: Always verify against the latest documentation
- Test Features: Actually test that features work as expected
- Update Both Places: Update both
endpoints
andfeatures
arrays - Document Limitations: Add any constraints to the
limitations
map
Example Update
# In provider_capabilities.ex
openai: %__MODULE__.ProviderInfo{
# ... other fields ...
endpoints: [
:chat, # /v1/chat/completions
:embeddings, # /v1/embeddings
:images, # /v1/images/generations
:audio, # /v1/audio/transcriptions, /v1/audio/translations, /v1/audio/speech
:moderations, # /v1/moderations
:files, # /v1/files
:fine_tuning, # /v1/fine-tuning/jobs
:assistants, # /v1/assistants
:threads, # /v1/threads
:runs, # /v1/threads/{thread_id}/runs
:vector_stores # /v1/vector_stores
],
features: [
:streaming,
:function_calling,
:vision,
:image_generation, # DALL-E
:speech_synthesis, # TTS
:speech_recognition, # Whisper
:embeddings,
:fine_tuning_api,
:assistants_api,
:code_interpreter,
:retrieval,
:structured_outputs,
:json_mode,
:logprobs,
:seed_control,
# ... etc
]
}
Testing Provider Capabilities
To verify capabilities are correctly configured:
# Check if a provider supports a feature
ExLLM.ProviderCapabilities.supports?(:openai, :image_generation)
# => true
# Find all providers that support embeddings
ExLLM.ProviderCapabilities.find_providers_with_features([:embeddings])
# => [:openai, :gemini, :bedrock, :ollama]
# Get full capability info
{:ok, caps} = ExLLM.ProviderCapabilities.get_capabilities(:openai)
IO.inspect(caps.features)
Common Gotchas
- Feature vs Endpoint: Some capabilities are features (
:vision
), others are endpoints (:images
) - Provider-Specific Names: Each provider may call the same feature differently
- Beta Features: Mark beta/experimental features in limitations
- Model-Specific: Some features depend on the specific model (e.g., vision only in GPT-4V)
- Region/Tier Limits: Some features have geographic or pricing tier restrictions
Contributing
When adding a new provider or updating capabilities:
- Research the provider's full API surface
- Test each capability you add
- Document any special requirements
- Update this guide if you discover new patterns