ExLLM.ResponseCache (ex_llm v0.5.0)

View Source

Response caching system for collecting and storing real provider responses.

This module allows ExLLM to cache actual responses from providers like OpenAI, Anthropic, OpenRouter, etc., and later replay them in tests using the Mock adapter.

Usage

Automatic Response Caching

Enable caching in your environment:

# Cache all responses during testing
export EX_LLM_CACHE_RESPONSES=true
export EX_LLM_CACHE_DIR="/path/to/cache/responses"

Manual Response Caching

# Store a response
ExLLM.ResponseCache.store_response("openai", request_key, response_data)

# Retrieve a cached response
cached = ExLLM.ResponseCache.get_response("openai", request_key)

# Load all responses for a provider
responses = ExLLM.ResponseCache.load_provider_responses("anthropic")

Integration with Mock Adapter

# Configure mock to use cached responses from OpenAI
ExLLM.ResponseCache.configure_mock_provider("openai")

# Use cached responses in tests
{:ok, response} = ExLLM.chat(messages, provider: :mock)

Cache Structure

Responses are stored in JSON files organized by provider:

cache/
 openai/
    chat_completions.json
    embeddings.json
    streaming.json
 anthropic/
    messages.json
    streaming.json
 openrouter/
     chat_completions.json
     models.json

Each file contains an array of cached request/response pairs with metadata.

Summary

Functions

Returns the cache directory path.

Returns true if response caching is enabled.

Clears all cached responses.

Clears all cached responses for a provider.

Configures the Mock adapter to use cached responses from a specific provider.

Retrieves a cached response matching the request.

Lists all available providers with cached responses.

Loads all cached responses for a provider.

Functions

cache_dir()

Returns the cache directory path.

caching_enabled?()

Returns true if response caching is enabled.

clear_all_cache()

Clears all cached responses.

clear_provider_cache(provider)

Clears all cached responses for a provider.

configure_mock_provider(provider)

Configures the Mock adapter to use cached responses from a specific provider.

get_response(provider, endpoint, request_data)

Retrieves a cached response matching the request.

list_cached_providers()

Lists all available providers with cached responses.

load_provider_responses(provider)

Loads all cached responses for a provider.

store_from_cache(cache_key, cached_response, provider, endpoint, request_metadata, disk_path \\ nil)

Stores a response from the unified cache system.

This is called by ExLLM.Cache when disk persistence is enabled.

store_response(provider, endpoint, request_data, response_data, opts \\ [])

Stores a response in the cache.