# Mix Tasks Guide ReqLLM provides powerful Mix tasks for text generation and model coverage validation. This guide covers all available tasks, their options, and common workflows. ## Overview ReqLLM includes these main Mix tasks: | Task | Alias | Purpose | |------|-------|---------| | `mix req_llm.doctor` | — | Diagnose installation and runtime configuration without provider calls | | `mix req_llm.migration_audit` | — | Find precise, mechanical V2 migration work without rewriting source | | `mix req_llm.gen` | `mix llm` | Generate text/objects from the command line | | `mix req_llm.model_compat` | `mix mc` | Validate model coverage with fixtures | | `mix req_llm.model_support` | — | Inspect and verify evidence-derived support tiers | | `mix req_llm.provider_drift` | — | Run bounded, read-only live anchor verification | ## mix req_llm.doctor Inspect the local ReqLLM runtime, provider registration, credential presence, model resolution, selected request surface, and Finch configuration. The default command performs no network probe or paid provider request and never prints credential values. ```bash mix req_llm.doctor mix req_llm.doctor --provider anthropic mix req_llm.doctor --model openai:gpt-4o-mini --operation chat mix req_llm.doctor --format json ``` Warnings such as missing optional credentials exit successfully. Invalid required configuration, an unknown requested provider or model, an unavailable selected surface, or an unhealthy Finch runtime produces a non-zero exit status. ### Machine-readable output Use `--format json` or `--json` for the stable schema-versioned result. Schema version 1 contains only redacted, JSON-safe data: ```json { "schema_version": 1, "status": "ok", "checks": [ { "id": "runtime.application", "layer": "runtime", "status": "ok", "message": "ReqLLM and its supervision tree are running.", "remediation": null, "details": {} } ] } ``` Top-level status is `ok`, `warning`, or `error`. Each check always includes `id`, `layer`, `status`, `message`, `remediation`, and `details`. New check IDs or detail fields may be added without changing schema version; existing common fields retain their meaning for schema version 1. ## mix req_llm.migration_audit Scan Elixir source for active ReqLLM deprecations and precise V2-readiness patterns. The task parses source without evaluating or rewriting it and makes no provider requests. ```bash mix req_llm.migration_audit mix req_llm.migration_audit lib test mix req_llm.migration_audit --exclude test/fixtures mix req_llm.migration_audit --format json ``` Actionable findings exit with status `1`; unreadable or invalid source exits with status `2`. Clean and advisory-only reports exit with status `0`. See the [V2 migration audit guide](v2-migration-audit.md) for the ledger schema, before/after examples, and static-analysis limitations. ## mix req_llm.gen Generate text or structured objects from any AI model with unified interface. Useful for testing, experimentation, and quick one-off generations. ### Basic Usage ```bash # Basic text generation mix req_llm.gen "Explain how neural networks work" # Streaming text mix req_llm.gen "Write a story about space" --stream # Generate JSON object mix req_llm.gen "Create a user profile for John Smith, age 30" --json # Specify model mix req_llm.gen "Hello!" --model anthropic:claude-3-5-sonnet ``` ### Options | Option | Alias | Default | Description | |--------|-------|---------|-------------| | `--model` | `-m` | Config or `openai:gpt-4o-mini` | Model specification (provider:model) | | `--stream` | `-s` | `true` | Stream the response token-by-token | | `--json` | `-j` | `false` | Generate structured JSON object | | `--temperature` | `-t` | Model default | Sampling temperature (0.0-2.0) | | `--log-level` | `-l` | `info` | Log verbosity: `debug`, `info`, `warning` | ### Examples #### Basic Text ```bash # Simple question mix req_llm.gen "What is 2+2?" # Multi-word prompt mix req_llm.gen "Explain quantum computing in simple terms" # With specific model mix req_llm.gen "Write a haiku" --model anthropic:claude-3-5-sonnet ``` #### Streaming ```bash # Stream basic text mix req_llm.gen "Tell me a joke" --stream # Stream with model selection mix req_llm.gen "Explain recursion" \ --model openai:gpt-4o \ --stream # Stream creative content mix req_llm.gen "Write a short story about AI" \ --stream \ --temperature 0.9 ``` #### JSON Generation ```bash # Generate structured object mix req_llm.gen "Create a profile for Jane Doe, software engineer" --json # With specific model mix req_llm.gen "Generate user data for Bob Smith" \ --model anthropic:claude-3-sonnet \ --json \ --temperature 0.1 # Stream JSON generation mix req_llm.gen "Generate a product listing" \ --json --stream \ --log-level debug ``` #### Log Levels ```bash # Quiet (only show generated content) mix req_llm.gen "What is 2+2?" --log-level warning # Normal (show model info and content) - default mix req_llm.gen "Hello!" --log-level info # Verbose (show timing and usage stats) mix req_llm.gen "Explain AI" --log-level debug ``` ### Built-in JSON Schema When using `--json`, a default "person" schema is used: ```json { "name": "string (required) - Full name of the person", "age": "integer - Age in years", "occupation": "string - Job or profession", "location": "string - City or region where they live" } ``` ### Output Examples #### Text Generation ``` openai:gpt-4o → "What is 2+2?" 4 250ms • 15→3 tokens • ~$0.000090 ``` #### Streaming ``` anthropic:claude-3-5-sonnet → "Write a haiku about coding" Code flows like water, Logic blooms in silent thought, Debug reveals truth. 850ms • 28→45 tokens • ~$0.000420 ``` #### JSON Generation ``` Generating object from openai:gpt-4o-mini Prompt: Create a user profile for John Smith, age 30 { "name": "John Smith", "age": 30, "occupation": "Software Engineer", "location": "San Francisco, CA" } ``` ### Configuration Set default model in `config/config.exs`: ```elixir config :req_llm, default_model: "openai:gpt-4o-mini" ``` ### Environment Variables API keys are required for each provider: ```bash export OPENAI_API_KEY=sk-... export ANTHROPIC_API_KEY=sk-ant-... export GOOGLE_API_KEY=... export OPENROUTER_API_KEY=... export XAI_API_KEY=... ``` ### Error Handling ```bash # Missing API key $ mix req_llm.gen "Hello" --model anthropic:claude-3-sonnet Error: API key not found for anthropic Please set your API key using one of these methods: 1. Environment variable: export ANTHROPIC_API_KEY=your-api-key 2. Application config: config :req_llm, anthropic_api_key: "your-api-key" 3. Pass directly: generate_text(model, prompt, api_key: "your-api-key") # Invalid model $ mix req_llm.gen "Hello" --model openai:gpt-invalid Error: Invalid model specification 'openai:gpt-invalid' Available openai models: • openai:gpt-4o • openai:gpt-4o-mini • openai:gpt-3.5-turbo ``` ## mix req_llm.model_compat (mix mc) Validate ReqLLM model coverage using fixture-based testing. This task is how ReqLLM ensures all "supported models" actually work. ### What It Does - Runs comprehensive tests against models from the registry - Validates capabilities (streaming, tool calling, object generation, etc.) - Uses cached fixtures for fast validation (no API calls by default) - Records live API responses when `--record` flag is used - Tracks validation state in `priv/supported_models.json` ### Basic Usage ```bash # Show models with passing fixtures mix req_llm.model_compat mix mc # List all models from registry mix mc --available # Test sample models mix mc --sample ``` ### Testing Specific Models ```bash # Test all Anthropic models mix mc anthropic mix mc "anthropic:*" # Test specific model mix mc "openai:gpt-4o" # Test all models from all providers mix mc "*:*" # Test by prefix mix mc "openai:gpt-4*" ``` ### Testing by Operation Type ```bash # Text generation models only (default) mix mc "google:*" --type text # Embedding models only mix mc "openai:*" --type embedding # Both text and embedding mix mc "google:*" --type all ``` ### Recording Fixtures **Important**: Recording makes live API calls and requires API keys. ```bash # Re-record fixtures for specific provider mix mc "xai:*" --record # Re-record specific model mix mc "openai:gpt-4o" --record # Force re-record all (ignores existing state) mix mc "*:*" --record-all ``` ### Sample Testing ```bash # Test sample subset (uses config/config.exs) mix mc --sample # Test sample for specific provider mix mc "anthropic:*" --sample # Configure samples in config/config.exs config :req_llm, sample_text_models: [ "openai:gpt-4o-mini", "anthropic:claude-3-5-haiku-20241022" ], sample_embedding_models: [ "openai:text-embedding-3-small" ] ``` ### Flags | Flag | Description | |------|-------------| | `--available` | List all models from registry (no filtering) | | `--sample` | Test sample subset from config | | `--type TYPE` | Filter by operation: `text`, `embedding`, or `all` | | `--record` | Re-record fixtures (live API calls) | | `--record-all` | Force re-record all fixtures (ignores state) | | `--debug` | Enable verbose fixture debugging output | ### Environment Variables ```bash # Set fixture mode REQ_LLM_FIXTURES_MODE=record mix mc # Select specific models REQ_LLM_MODELS="anthropic:*" mix mc # Set operation type REQ_LLM_OPERATION=embedding mix mc # Enable debug output REQ_LLM_DEBUG=1 mix mc # Include full API responses REQ_LLM_INCLUDE_RESPONSES=1 mix mc ``` ### How Validation Works The task validates models through comprehensive capability-focused tests: 1. **Basic generate_text** (non-streaming) - All models 2. **Streaming** with system context + creative params 3. **Token limit constraints** 4. **Usage metrics and cost calculations** 5. **Tool calling** - Multi-tool selection and refusal 6. **Object generation** (streaming and non-streaming) 7. **Reasoning/thinking tokens** (GPT-5, Claude) Each test that passes creates a fixture file in `test/support/fixtures///`. ### Fixture State Tracking The `priv/supported_models.json` file tracks validation status: ```json { "anthropic:claude-3-5-sonnet-20241022": { "status": "pass", "last_checked": "2025-01-29T10:30:00Z" }, "openai:gpt-4o": { "status": "pass", "last_checked": "2025-01-29T10:30:15Z" }, "xai:grok-vision-beta": { "status": "excluded", "last_checked": null } } ``` Scenario-level evidence is stored separately in `priv/model_compat_scenarios.json`. That versioned artifact records the model, exact execution surface inferred from the fixture request URL, scenario, proof level, observations, fixture names, and classified failure layer. It is used only by compatibility tooling and generated documentation; it never changes model resolution or request routing. ## `mix req_llm.model_support` Inspect the conservative support tiers derived from current scenario evidence: ```bash mix req_llm.model_support mix req_llm.model_support --model openai:gpt-4o-mini mix req_llm.model_compat --available --sample --support-tiers ``` Maintainers generate and verify the checked-in evidence reference with: ```bash mix req_llm.model_support --generate mix req_llm.model_support --check ``` The generated [`model-support.md`](model-support.md) reference is deterministic for a given catalog and evidence snapshot. Tiers are surface-specific: - `first_class` requires current passing evidence for the entire operation baseline; - `best_effort` requires at least one current baseline pass but is incomplete; - `experimental` means evidence or current catalog declaration is missing or stale; and - `unsupported` includes an explicit reason such as a baseline failure layer or an operation absent from current model metadata. Catalog presence alone is never evidence. These tiers do not act as a runtime allowlist. ## `mix req_llm.provider_drift` Validate the sparse live anchor configuration without making provider calls: ```bash mix req_llm.provider_drift --dry-run ``` Run every anchor whose credential is available, or select a provider: ```bash mix req_llm.provider_drift mix req_llm.provider_drift --provider openai mix req_llm.provider_drift --provider anthropic,google ``` Write the sanitized JSON and Markdown reports to a chosen directory: ```bash mix req_llm.provider_drift --output-dir .artifacts/provider-drift ``` | Option | Purpose | | --- | --- | | `--dry-run` | Validate anchors, evidence references, and guardrails without provider requests | | `--provider NAME[,NAME]` | Limit the run to selected configured providers | | `--config PATH` | Validate and run a different anchor configuration | | `--output-dir PATH` | Choose the report directory; defaults to `_build/provider_drift` | The default matrix is `priv/provider_drift_anchors.json`. It explicitly caps anchor count, concurrency, per-anchor timeout, output tokens, and estimated cost. Missing credentials produce skipped results rather than failures. An executed failure uses the same resolution, planning, encoding, transport, decoding, materialization, assertion, and provider-drift layers as compatibility evidence. The task records provider responses only into a temporary directory, derives the actual execution surface, and deletes the temporary files. Reports contain no prompts, response bodies, or credential values. They embed completed results in the versioned compatibility evidence schema with mode `live_probe`, but do not modify the checked-in evidence or influence runtime support. Use the separate `mix req_llm.model_compat MODEL --scenario SCENARIO --record` action when fixture recording is intentional. ### Example Output ``` ---------------------------------------------------- Model Coverage Status ---------------------------------------------------- Anthropic ✓ claude-3-5-sonnet-20241022 (flagship) ✓ claude-3-5-haiku-20241022 (fast) ✗ claude-3-opus-20240229 (flagship) 2 pass, 1 fail, 0 excluded, 0 untested | 66.7% coverage OpenAI ✓ gpt-4o (flagship) ✓ gpt-4o-mini (fast) ✓ gpt-3.5-turbo (fast) 3 pass, 0 fail, 0 excluded, 0 untested | 100.0% coverage Overall Coverage: 5/6 models validated (83.3%) ``` ### Testing Workflow #### Adding a New Provider ```bash # 1. Implement provider module # 2. Create test file using Comprehensive macro # 3. Record initial fixtures mix mc "newprovider:*" --record # 4. Verify all tests pass mix mc "newprovider" ``` #### Updating Model Coverage ```bash # 1. Update llm_db for latest model metadata mix deps.update llm_db # 2. Record fixtures for new models mix mc "openai:gpt-5-mini" --record # 3. Validate updated coverage mix mc openai ``` #### Refreshing Fixtures ```bash # Refresh specific provider (checks for API changes) mix mc "anthropic:*" --record # Refresh all (expensive, requires all API keys) mix mc "*:*" --record ``` See [Fixture Testing Guide](fixture-testing.md) for complete details. ## Common Workflows ### Initial Setup ```bash # 1. Validate sample models mix mc --sample # 2. Test a quick generation mix req_llm.gen "Hello, world!" --model openai:gpt-4o-mini ``` ### Before Deploying ```bash # 1. Validate all models mix mc "*:*" # 2. Check coverage report mix mc ``` ### Adding a New Provider ```bash # 1. Add models to local patch cat > priv/models_local/newprovider.json <