Testing Guide
View SourceExLLM includes a comprehensive testing system with intelligent caching, semantic tagging, and 24 specialized Mix aliases for targeted test execution.
Quick Start
# Run all tests (fast - uses cache when available)
mix test
# Run provider-specific tests
mix test.anthropic
mix test.openai
mix test.gemini
# Run integration tests with live APIs
mix test.integration --include live_api
# Run tests by capability
mix test.streaming
mix test.vision
mix test.oauth2
# Manage test cache
mix ex_llm.cache stats
mix ex_llm.cache clean --older-than 7d
Test Organization
Test Tags
ExLLM uses semantic tags to organize tests by requirements, capabilities, and providers:
Requirement Tags
:requires_api_key
- Tests needing API keys with automatic provider detection:requires_oauth
- Tests needing OAuth2 authentication (e.g., Gemini APIs):requires_service
- Tests needing local services (Ollama, LM Studio):requires_resource
- Tests needing pre-existing resources (tuned models, corpora)
Test Type Tags
:live_api
- Tests that call live provider APIs:integration
- Integration tests with external services:external
- Tests making external network calls:unit
- Unit tests (isolated, no external dependencies)
Provider Tags
:anthropic
,:openai
,:gemini
,:groq
,:mistral
:openrouter
,:perplexity
,:ollama
,:lmstudio
,:bumblebee
Capability Tags
:streaming
- Tests for streaming responses:vision
- Tests for image/vision capabilities:multimodal
- Tests for multimodal inputs:function_calling
- Tests for tool/function calling:embedding
- Tests for embedding generation
Mix Test Aliases
ExLLM provides 24 specialized test aliases for targeted execution:
Provider-Specific Tests
# Test individual providers
mix test.anthropic # Anthropic Claude tests
mix test.openai # OpenAI GPT tests
mix test.gemini # Google Gemini tests
mix test.groq # Groq tests
mix test.mistral # Mistral AI tests
mix test.openrouter # OpenRouter tests
mix test.perplexity # Perplexity tests
mix test.ollama # Ollama local tests
mix test.lmstudio # LM Studio tests
mix test.bumblebee # Bumblebee local tests
Test Type Aliases
# By test type
mix test.unit # Unit tests only
mix test.integration # Integration tests
mix test.external # Tests with external calls
mix test.oauth2 # OAuth2 authentication tests
Capability-Based Tests
# By capability
mix test.streaming # Streaming response tests
mix test.vision # Vision/image processing tests
mix test.multimodal # Multimodal input tests
mix test.function_calling # Function/tool calling tests
mix test.embedding # Embedding generation tests
Environment-Based Tests
# By environment needs
mix test.live_api # Tests calling live APIs
mix test.local_only # Local-only tests (no API calls)
mix test.fast # Fast tests (cached/mocked)
mix test.all # All tests including slow ones
Test Caching System
ExLLM includes an advanced caching system that provides 25x speed improvements for integration tests.
How It Works
- Automatic Detection: Tests tagged with
:live_api
are automatically cached - Smart Exclusions: Destructive operations (create, delete, modify) are not cached
- TTL Management: Cached responses expire after 7 days by default
- Fallback Strategies: Multiple matching algorithms for cache hits
Cache Management
# View cache statistics
mix ex_llm.cache stats
# Clean old cache entries
mix ex_llm.cache clean --older-than 7d
# Clear all cache
mix ex_llm.cache clear
# Show cache details for a provider
mix ex_llm.cache show anthropic
Configuration
Configure caching via environment variables:
# Enable/disable caching
export EX_LLM_TEST_CACHE_ENABLED=true
# Cache directory
export EX_LLM_TEST_CACHE_DIR="test/cache"
# TTL for cached responses (in seconds)
export EX_LLM_TEST_CACHE_TTL=604800 # 7 days
# Cache destructive operations
export EX_LLM_TEST_CACHE_DESTRUCTIVE_OPS=false
Writing Tests
Basic Test Structure
defmodule MyProviderTest do
use ExUnit.Case
# Module-level tags
@moduletag :integration
@moduletag :live_api
@moduletag :requires_api_key
@moduletag provider: :anthropic
# Import cache helpers
import ExLLM.TestCacheHelpers
setup_all do
enable_cache_debug()
:ok
end
setup context do
setup_test_cache(context)
on_exit(fn -> ExLLM.TestCacheDetector.clear_test_context() end)
:ok
end
test "basic chat completion" do
{:ok, response} = ExLLM.chat(:anthropic, [
%{role: "user", content: "Hello!"}
])
assert response.content != ""
end
end
Using ExLLM.Case for Automatic Requirements
defmodule MyProviderTest do
use ExLLM.Case, async: true
@moduletag :requires_api_key
@moduletag provider: :openai
test "test with automatic API key checking", context do
# Automatically skips if OPENAI_API_KEY not set
check_test_requirements!(context)
{:ok, response} = ExLLM.chat(:openai, [
%{role: "user", content: "Test"}
])
assert response.content != ""
end
end
OAuth2 Tests
defmodule GeminiOAuth2Test do
use ExLLM.Case, async: true
@moduletag :requires_oauth
@moduletag provider: :gemini
test "OAuth2 API call", context do
check_test_requirements!(context)
# OAuth token automatically provided if available
oauth_token = get_oauth_token(context)
{:ok, response} = ExLLM.Gemini.Permissions.list_permissions(
"tunedModels/test",
oauth_token: oauth_token
)
assert is_list(response.permissions)
end
end
Test Exclusions
Default Exclusions
By default, these tests are excluded unless explicitly included:
# In test_helper.exs
ExUnit.configure(exclude: [
:live_api, # Exclude live API calls by default
:requires_api_key, # Exclude tests needing API keys
:requires_oauth, # Exclude OAuth tests
:requires_service, # Exclude tests needing local services
:integration, # Exclude integration tests
:external # Exclude external network tests
])
Running Excluded Tests
# Include specific tags
mix test --include live_api
mix test --include requires_api_key
mix test --include oauth2
# Include multiple tags
mix test --include live_api --include streaming
# Run only specific tags
mix test --only provider:anthropic
mix test --only streaming
CI/CD Integration
GitHub Actions Example
name: Tests
on: [push, pull_request]
jobs:
unit-tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: erlef/setup-beam@v1
with:
elixir-version: '1.18'
otp-version: '28'
# Unit tests only (no API keys needed)
- run: mix test.unit
integration-tests:
runs-on: ubuntu-latest
if: github.event_name == 'push'
steps:
- uses: actions/checkout@v3
- uses: erlef/setup-beam@v1
# Integration tests with caching
- run: mix test.integration --include live_api
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
EX_LLM_TEST_CACHE_ENABLED: true
Performance Tips
- Use Caching: Enable test caching for 25x faster integration tests
- Tag Appropriately: Use semantic tags for precise test selection
- Run Targeted Tests: Use Mix aliases to run only what you need
- Cache Management: Regularly clean old cache entries
- Local Services: Use Ollama/LM Studio for development without API costs
Troubleshooting
Common Issues
Tests skipped with "API key required":
# Set the required API key
export ANTHROPIC_API_KEY="your-key"
mix test.anthropic
OAuth2 tests failing:
# Setup OAuth2 first
elixir scripts/setup_oauth2.exs
# Then refresh token
elixir scripts/refresh_oauth2_token.exs
Cache not working:
# Check cache configuration
mix ex_llm.cache stats
# Enable debug logging
export EX_LLM_LOG_LEVEL=debug
Tests timing out:
# Use cached responses
export EX_LLM_TEST_CACHE_ENABLED=true
mix test --include live_api
Debug Logging
Enable debug logging to troubleshoot test issues:
export EX_LLM_LOG_LEVEL=debug
export EX_LLM_LOG_COMPONENTS=http_client,cache,test_detector
mix test --include live_api
This comprehensive testing system ensures reliable, fast, and well-organized tests across all ExLLM functionality.