Testing Guide

View Source

ExLLM includes a comprehensive testing system with intelligent caching, semantic tagging, and 24 specialized Mix aliases for targeted test execution.

Quick Start

# Run all tests (fast - uses cache when available)
mix test

# Run provider-specific tests
mix test.anthropic
mix test.openai
mix test.gemini

# Run integration tests with live APIs
mix test.integration --include live_api

# Run tests by capability
mix test.streaming
mix test.vision
mix test.oauth2

# Manage test cache
mix ex_llm.cache stats
mix ex_llm.cache clean --older-than 7d

Test Organization

Test Tags

ExLLM uses semantic tags to organize tests by requirements, capabilities, and providers:

Requirement Tags

  • :requires_api_key - Tests needing API keys with automatic provider detection
  • :requires_oauth - Tests needing OAuth2 authentication (e.g., Gemini APIs)
  • :requires_service - Tests needing local services (Ollama, LM Studio)
  • :requires_resource - Tests needing pre-existing resources (tuned models, corpora)

Test Type Tags

  • :live_api - Tests that call live provider APIs
  • :integration - Integration tests with external services
  • :external - Tests making external network calls
  • :unit - Unit tests (isolated, no external dependencies)

Provider Tags

  • :anthropic, :openai, :gemini, :groq, :mistral
  • :openrouter, :perplexity, :ollama, :lmstudio, :bumblebee

Capability Tags

  • :streaming - Tests for streaming responses
  • :vision - Tests for image/vision capabilities
  • :multimodal - Tests for multimodal inputs
  • :function_calling - Tests for tool/function calling
  • :embedding - Tests for embedding generation

Mix Test Aliases

ExLLM provides 24 specialized test aliases for targeted execution:

Provider-Specific Tests

# Test individual providers
mix test.anthropic       # Anthropic Claude tests
mix test.openai          # OpenAI GPT tests  
mix test.gemini          # Google Gemini tests
mix test.groq            # Groq tests
mix test.mistral         # Mistral AI tests
mix test.openrouter      # OpenRouter tests
mix test.perplexity      # Perplexity tests
mix test.ollama          # Ollama local tests
mix test.lmstudio        # LM Studio tests
mix test.bumblebee       # Bumblebee local tests

Test Type Aliases

# By test type
mix test.unit            # Unit tests only
mix test.integration     # Integration tests
mix test.external        # Tests with external calls
mix test.oauth2          # OAuth2 authentication tests

Capability-Based Tests

# By capability
mix test.streaming       # Streaming response tests
mix test.vision          # Vision/image processing tests
mix test.multimodal      # Multimodal input tests
mix test.function_calling # Function/tool calling tests
mix test.embedding       # Embedding generation tests

Environment-Based Tests

# By environment needs
mix test.live_api        # Tests calling live APIs
mix test.local_only      # Local-only tests (no API calls)
mix test.fast            # Fast tests (cached/mocked)
mix test.all             # All tests including slow ones

Test Caching System

ExLLM includes an advanced caching system that provides 25x speed improvements for integration tests.

How It Works

  1. Automatic Detection: Tests tagged with :live_api are automatically cached
  2. Smart Exclusions: Destructive operations (create, delete, modify) are not cached
  3. TTL Management: Cached responses expire after 7 days by default
  4. Fallback Strategies: Multiple matching algorithms for cache hits

Cache Management

# View cache statistics
mix ex_llm.cache stats

# Clean old cache entries  
mix ex_llm.cache clean --older-than 7d

# Clear all cache
mix ex_llm.cache clear

# Show cache details for a provider
mix ex_llm.cache show anthropic

Configuration

Configure caching via environment variables:

# Enable/disable caching
export EX_LLM_TEST_CACHE_ENABLED=true

# Cache directory
export EX_LLM_TEST_CACHE_DIR="test/cache"

# TTL for cached responses (in seconds)
export EX_LLM_TEST_CACHE_TTL=604800  # 7 days

# Cache destructive operations
export EX_LLM_TEST_CACHE_DESTRUCTIVE_OPS=false

Writing Tests

Basic Test Structure

defmodule MyProviderTest do
  use ExUnit.Case
  
  # Module-level tags
  @moduletag :integration
  @moduletag :live_api
  @moduletag :requires_api_key
  @moduletag provider: :anthropic
  
  # Import cache helpers
  import ExLLM.TestCacheHelpers
  
  setup_all do
    enable_cache_debug()
    :ok
  end
  
  setup context do
    setup_test_cache(context)
    on_exit(fn -> ExLLM.TestCacheDetector.clear_test_context() end)
    :ok
  end
  
  test "basic chat completion" do
    {:ok, response} = ExLLM.chat(:anthropic, [
      %{role: "user", content: "Hello!"}
    ])
    
    assert response.content != ""
  end
end

Using ExLLM.Case for Automatic Requirements

defmodule MyProviderTest do
  use ExLLM.Case, async: true
  
  @moduletag :requires_api_key
  @moduletag provider: :openai
  
  test "test with automatic API key checking", context do
    # Automatically skips if OPENAI_API_KEY not set
    check_test_requirements!(context)
    
    {:ok, response} = ExLLM.chat(:openai, [
      %{role: "user", content: "Test"}
    ])
    
    assert response.content != ""
  end
end

OAuth2 Tests

defmodule GeminiOAuth2Test do
  use ExLLM.Case, async: true
  
  @moduletag :requires_oauth
  @moduletag provider: :gemini
  
  test "OAuth2 API call", context do
    check_test_requirements!(context)
    
    # OAuth token automatically provided if available
    oauth_token = get_oauth_token(context)
    
    {:ok, response} = ExLLM.Gemini.Permissions.list_permissions(
      "tunedModels/test",
      oauth_token: oauth_token
    )
    
    assert is_list(response.permissions)
  end
end

Test Exclusions

Default Exclusions

By default, these tests are excluded unless explicitly included:

# In test_helper.exs
ExUnit.configure(exclude: [
  :live_api,           # Exclude live API calls by default
  :requires_api_key,   # Exclude tests needing API keys
  :requires_oauth,     # Exclude OAuth tests
  :requires_service,   # Exclude tests needing local services
  :integration,        # Exclude integration tests
  :external           # Exclude external network tests
])

Running Excluded Tests

# Include specific tags
mix test --include live_api
mix test --include requires_api_key
mix test --include oauth2

# Include multiple tags
mix test --include live_api --include streaming

# Run only specific tags
mix test --only provider:anthropic
mix test --only streaming

CI/CD Integration

GitHub Actions Example

name: Tests
on: [push, pull_request]

jobs:
  unit-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: erlef/setup-beam@v1
        with:
          elixir-version: '1.18'
          otp-version: '28'
      
      # Unit tests only (no API keys needed)
      - run: mix test.unit
  
  integration-tests:
    runs-on: ubuntu-latest
    if: github.event_name == 'push'
    steps:
      - uses: actions/checkout@v3
      - uses: erlef/setup-beam@v1
      
      # Integration tests with caching
      - run: mix test.integration --include live_api
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          EX_LLM_TEST_CACHE_ENABLED: true

Performance Tips

  1. Use Caching: Enable test caching for 25x faster integration tests
  2. Tag Appropriately: Use semantic tags for precise test selection
  3. Run Targeted Tests: Use Mix aliases to run only what you need
  4. Cache Management: Regularly clean old cache entries
  5. Local Services: Use Ollama/LM Studio for development without API costs

Troubleshooting

Common Issues

Tests skipped with "API key required":

# Set the required API key
export ANTHROPIC_API_KEY="your-key"
mix test.anthropic

OAuth2 tests failing:

# Setup OAuth2 first
elixir scripts/setup_oauth2.exs
# Then refresh token
elixir scripts/refresh_oauth2_token.exs

Cache not working:

# Check cache configuration
mix ex_llm.cache stats
# Enable debug logging
export EX_LLM_LOG_LEVEL=debug

Tests timing out:

# Use cached responses
export EX_LLM_TEST_CACHE_ENABLED=true
mix test --include live_api

Debug Logging

Enable debug logging to troubleshoot test issues:

export EX_LLM_LOG_LEVEL=debug
export EX_LLM_LOG_COMPONENTS=http_client,cache,test_detector
mix test --include live_api

This comprehensive testing system ensures reliable, fast, and well-organized tests across all ExLLM functionality.