# GEPA Live Examples All examples in this directory are live-only. They call the real selected LLM backend through the GEPA LLM facade. Deterministic replacement modules are limited to tests. The example-only CLI helper lives in `examples/support/live_cli.exs`. It is intentionally not part of `lib/` or the public package API. ## Fast Start If `GEMINI_API_KEY`, `GOOGLE_API_KEY`, `OPENAI_API_KEY`, or `ANTHROPIC_API_KEY` is configured, the fastest live smoke is: ```bash mix run examples/05_llm_adapters.exs -- --simple ``` To run every example with small built-in demo prompts/data: ```bash examples/run_all.sh --simple ``` Both commands make real LLM calls. They may incur provider costs. ## Cost Warning Each example prints a live-call warning before making any LLM call. `examples/run_all.sh` prints a stronger warning because it executes the full example suite and can make many calls. ## Sensible Defaults When `--adapter` and `--provider` are omitted, examples default to ReqLLM using the first configured hosted provider in this order: - Gemini: `GEMINI_API_KEY`, then `GOOGLE_API_KEY` - OpenAI: `OPENAI_API_KEY` - Anthropic: `ANTHROPIC_API_KEY` Explicit CLI options override these defaults: - `--adapter req_llm|asm` - `--provider openai|gemini|anthropic` for ReqLLM - `--provider codex|codex_exec|claude|gemini|amp` for ASM - `--api-key VALUE` for an explicit ReqLLM key override - `--model VALUE` for an explicit model override If `--adapter asm` is provided without `--provider`, the default ASM provider is `gemini`. Default models: - `--adapter req_llm --provider openai` uses `gpt-5.4-mini` - `--adapter req_llm --provider gemini` uses `gemini-3.1-flash-lite-preview` - `--adapter req_llm --provider anthropic` uses `claude-haiku-4-5` - `--adapter asm --provider codex` uses `gpt-5.4-mini` - `--adapter asm --provider gemini` uses `gemini-3.1-flash-lite-preview` - `--adapter asm --provider claude` uses the ASM/Claude default unless `--model` is provided - `--adapter asm --provider amp` uses the ASM/Amp default unless `--model` is provided ## ReqLLM ReqLLM examples support OpenAI, Gemini, and Anthropic: ```bash mix run examples/05_llm_adapters.exs -- \ --provider gemini \ --input "Reply with exactly: gepa adapter ok" ``` Use `--api-key` only when you want to override the default key lookup: ```bash mix run examples/05_llm_adapters.exs -- \ --adapter req_llm \ --provider openai \ --api-key sk-... \ --input "Reply with exactly: gepa adapter ok" ``` Structured output is available where the selected ReqLLM provider supports it: ```bash mix run examples/05_llm_adapters.exs -- \ --provider gemini \ --structured-output \ --input "Return an improved instruction for a concise QA assistant." ``` ## Agent Session Manager ASM examples support local/CLI providers through `../agent_session_manager`: - `codex` - `codex_exec` - `claude` - `gemini` - `amp` Codex through ASM: ```bash mix run examples/05_llm_adapters.exs -- \ --adapter asm \ --provider codex \ --lane core \ --session gepa_codex_smoke \ --input "Reply with exactly: gepa adapter ok" ``` ASM streaming emits text chunks at the GEPA facade boundary: ```bash mix run examples/05_llm_adapters.exs -- \ --adapter asm \ --provider codex \ --lane core \ --session gepa_stream \ --stream \ --input "Stream a short status update." ``` ASM provider prerequisites, authentication, sandboxing, and local CLI setup are managed outside GEPA by Agent Session Manager and the selected CLI provider. ## Simple Mode `--simple` is a live convenience mode: - It still makes real LLM calls. - It infers the provider from configured keys unless you pass explicit provider options. - It supplies small built-in demo prompts/data for examples that otherwise need JSONL input. - It uses a small optimization budget by default: `--max-metric-calls 2` and `--minibatch-size 1`. Examples: ```bash mix run examples/01_quick_start.exs -- --simple mix run examples/02_math_problems.exs -- --simple mix run examples/03_custom_adapter.exs -- --simple mix run examples/04_state_persistence.exs -- --simple mix run examples/05_llm_adapters.exs -- --simple mix run examples/06_stop_conditions.exs -- --simple mix run examples/07_merge_and_policies.exs -- --simple mix run examples/08_optimize_anything_single_task.exs -- --simple mix run examples/09_optimize_anything_dataset.exs -- --simple mix run examples/10_code_execution.exs -- --simple mix run examples/11_refiner.exs -- --simple mix run examples/12_tracking.exs -- --simple mix run examples/13_adrs_cloud_optimization.exs -- --simple mix run examples/14_arc_grid.exs -- --simple mix run examples/15_blackbox_search.exs -- --simple mix run examples/16_circle_packing.exs -- --simple mix run examples/17_qdrant_rag.exs -- --simple mix run examples/18_confidence_adapter.exs -- --simple ``` You can combine `--simple` with explicit backend selection: ```bash mix run examples/05_llm_adapters.exs -- \ --simple \ --adapter asm \ --provider codex \ --lane core \ --session gepa_simple_codex ``` ## Data Files Without `--simple`, optimization examples require JSONL data that you provide. Each line must be a JSON object. Question/answer examples use: ```json {"input":"","answer":""} ``` Sentiment adapter example uses: ```json {"text":"","sentiment":"positive"} ``` Supported sentiment labels are `positive`, `negative`, and `neutral`. ## Examples ### 01 Quick Start Runs a small live GEPA optimization over question/answer JSONL data. ```bash mix run examples/01_quick_start.exs -- \ --provider gemini \ --train-jsonl /path/to/qa_train.jsonl \ --val-jsonl /path/to/qa_val.jsonl \ --max-metric-calls 8 ``` ### 02 Math Problems Runs a live GEPA optimization over AIME-style math question/answer JSONL data. The solver prompt is optimized while the selected live adapter answers each problem. Rows should contain `input` and a final numeric `answer`; `--simple` uses a tiny built-in arithmetic set for a quick live smoke. ```bash mix run examples/02_math_problems.exs -- \ --provider gemini \ --train-jsonl /path/to/math_train.jsonl \ --val-jsonl /path/to/math_val.jsonl \ --max-metric-calls 8 ``` ### 03 Custom Sentiment Adapter Runs a custom live adapter over `text`/`sentiment` JSONL data. ```bash mix run examples/03_custom_adapter.exs -- \ --adapter asm \ --provider codex \ --lane core \ --session gepa_sentiment \ --train-jsonl /path/to/sentiment_train.jsonl \ --val-jsonl /path/to/sentiment_val.jsonl \ --max-metric-calls 8 ``` ### 04 State Persistence Runs or resumes a live optimization with checkpoint persistence. ```bash mix run examples/04_state_persistence.exs -- \ --provider gemini \ --train-jsonl /path/to/persistence_train.jsonl \ --val-jsonl /path/to/persistence_val.jsonl \ --run-dir /path/to/gepa_run \ --max-metric-calls 8 ``` ### 05 LLM Adapter Smoke Makes one real completion call through ReqLLM or ASM. ```bash mix run examples/05_llm_adapters.exs -- --simple ``` ```bash mix run examples/05_llm_adapters.exs -- \ --adapter asm \ --provider codex \ --lane core \ --session gepa_adapter_smoke \ --input "Reply with exactly: gepa adapter ok" ``` ### 06 Stop Conditions Runs GEPA with multiple stop conditions composed together. ```bash mix run examples/06_stop_conditions.exs -- --simple ``` ### 07 Merge and Policies Runs GEPA with merge enabled and explicit strategy settings. ```bash mix run examples/07_merge_and_policies.exs -- --simple ``` ### 08 Optimize Anything Single Task Optimizes a single prompt string with a live LLM-backed evaluator. ```bash mix run examples/08_optimize_anything_single_task.exs -- --simple ``` ### 09 Optimize Anything Dataset Optimizes a prompt map over question/answer examples. ```bash mix run examples/09_optimize_anything_dataset.exs -- --simple ``` ### 10 Code Execution Optimizes a small Elixir snippet and evaluates it through `GEPA.CodeExecution`. ```bash mix run examples/10_code_execution.exs -- --simple ``` ### 11 Refiner Runs `GEPA.OptimizeAnything` with the per-evaluation refiner enabled. ```bash mix run examples/11_refiner.exs -- --simple ``` ### 12 Tracking Runs GEPA with `GEPA.Tracking.InMemory` enabled. ```bash mix run examples/12_tracking.exs -- --simple ``` ### 13 ADR Cloud Optimization Optimizes a cloud architecture-decision policy across two headless ADR-style domains: deadline-aware spot/on-demand scheduling and cost-aware multi-cloud broadcast routing. The evaluator uses built-in scenario maps and the selected live LLM adapter proposes policy revisions through `GEPA.OptimizeAnything`. ```bash mix run examples/13_adrs_cloud_optimization.exs -- --simple ``` Expected output includes `ADR Cloud Optimization Complete`, the best validation score, and the best policy text. If the score stays low, increase `--max-metric-calls` or use a stronger reflection model. ### 14 ARC Grid Optimizes a headless ARC grid-solving instruction. The live adapter sees training input/output grids and predicts a test output grid as compact JSON. `--simple` uses a tiny fixture set; custom JSONL rows should provide `input` as the full ARC task prompt and `answer` as the exact JSON grid to match. ```bash mix run examples/14_arc_grid.exs -- --simple ``` Expected output includes `ARC Grid Optimization Complete`, the best validation score, and the best instruction. If exact JSON matching is brittle for a provider, make the `answer` string and task prompt agree on compact formatting. ### 15 Blackbox Search Optimizes executable Elixir search code for a small blackbox minimization problem. The evaluator executes candidate code with `GEPA.CodeExecution`, tracks objective-call budget use, and feeds prior best trials back into later proposals. ```bash mix run examples/15_blackbox_search.exs -- --simple ``` Expected output includes `Blackbox Search Optimization Complete`, the best transformed score, and the best candidate code. If generated code fails, the feedback includes execution errors and budget details. ### 16 Circle Packing Optimizes executable Elixir geometry code that packs circles inside the unit square. The evaluator runs candidate code with `GEPA.CodeExecution`, validates the returned circle list, scores valid layouts by the sum of radii, and feeds the best prior layout back into later candidates. ```bash mix run examples/16_circle_packing.exs -- --simple ``` Candidate code must bind `pack = fn config, current_best_solution -> ... end` and return `%{circles: circles, all_scores: scores}`. Each circle may be `%{x: x, y: y, r: r}` or `[x, y, r]`. `--simple` uses four circles for a short live smoke; full mode uses twenty-six circles to match the upstream problem shape. Expected output includes `Circle Packing Optimization Complete`, the problem size, the best `sum_radii`, and the best candidate code. If generated code returns an invalid layout, evaluator feedback reports shape errors, boundary violations, negative radii, overlaps, stdout, and execution errors. ### 17 Qdrant RAG Runs Generic RAG with a real local Qdrant container, real ReqLLM/Gemini embeddings, and real Agent Session Manager/Gemini inference. Qdrant stores and searches vectors; it does not create embeddings. Start Qdrant first: ```bash docker compose up -d qdrant ``` Then run the live integration smoke: ```bash mix run examples/17_qdrant_rag.exs -- --simple ``` The default inference path is `--adapter asm --provider gemini` using `gemini-3.1-flash-lite-preview`. The default embedding model is `google:gemini-embedding-001` through ReqLLM. Override local services with: ```bash mix run examples/17_qdrant_rag.exs -- \ --adapter asm \ --provider gemini \ --qdrant-url http://localhost:6333 \ --collection gepa_ex_qdrant_rag \ --embedding-model gemini-embedding-001 \ --max-metric-calls 2 ``` ### 18 Confidence Adapter Runs live structured-output classification through `GEPA.Adapters.Confidence`. The example uses ReqLLM structured output so the model returns a JSON object with `category`. Provider logprobs are used when available; when the selected provider does not expose logprobs, confidence probability remains explicitly unavailable rather than being synthesized. ```bash mix run examples/18_confidence_adapter.exs -- --simple ``` Custom JSONL rows should contain `input` and `answer`, where `answer` is one of the categories described by your prompt. The built-in simple mode uses `billing`, `shipping`, and `product`. Expected output includes `Confidence Adapter Optimization Complete`, the structured-output path, the confidence-source policy, the best score, and the best prompt. Use ReqLLM for this example; ASM structured output is intentionally unsupported until ASM exposes a native structured-output contract. ## Run Everything `run_all.sh` runs every example. Fast live demo: ```bash examples/run_all.sh --simple ``` With your own data: ```bash examples/run_all.sh \ --provider gemini \ --qa-train-jsonl /path/to/qa_train.jsonl \ --qa-val-jsonl /path/to/qa_val.jsonl \ --math-train-jsonl /path/to/math_train.jsonl \ --math-val-jsonl /path/to/math_val.jsonl \ --sentiment-train-jsonl /path/to/sentiment_train.jsonl \ --sentiment-val-jsonl /path/to/sentiment_val.jsonl \ --persistence-train-jsonl /path/to/persistence_train.jsonl \ --persistence-val-jsonl /path/to/persistence_val.jsonl \ --run-dir /path/to/gepa_run_all \ --smoke-input "Reply with exactly: gepa adapter ok" \ --max-metric-calls 8 ``` ASM Codex: ```bash examples/run_all.sh \ --adapter asm \ --provider codex \ --lane core \ --session gepa_run_all \ --simple ``` ## Help Every example and the runner support `--help`: ```bash mix run examples/01_quick_start.exs -- --help mix run examples/05_llm_adapters.exs -- --help examples/run_all.sh --help ``` Calling `examples/run_all.sh` without arguments prints the same help menu and does not run the suite. Calling an individual example without required args prints help plus concrete missing-argument errors and exits before any LLM call. ## Troubleshooting - Missing default key: set `GEMINI_API_KEY`, `GOOGLE_API_KEY`, `OPENAI_API_KEY`, or `ANTHROPIC_API_KEY`, or pass `--adapter/--provider/--api-key`. - Wrong literal API key: pass the actual secret value, for example `--api-key "$GEMINI_API_KEY"`, not `--api-key GEMINI_API_KEY`. - ReqLLM provider failure: confirm the hosted provider key and selected model are valid. - ASM provider failure: confirm the local provider CLI works outside GEPA and that the selected `--lane` is valid. - Streaming failure: use `--adapter asm`; ReqLLM streaming is intentionally not exposed by this temporary facade. - Structured output failure with ASM: use ReqLLM for structured output until ASM has a native structured-output contract. - JSONL load failure: verify each line is valid JSON and uses the fields required by the selected example. - Qdrant RAG failure: run `docker compose up -d qdrant`, confirm `curl http://localhost:6333/collections`, and set `GEMINI_API_KEY` or `GOOGLE_API_KEY` for embeddings.