Live LLM Tool Loop

Copy Markdown View Source

Use this guide to verify a real provider call and one complete model-operation loop. Keep this check separate from the deterministic test suite because it uses credentials, network access, provider quota, and a non-deterministic model.

Prerequisites

Complete Getting Started and Tools And Operations first.

You need:

  • a supported model in Jidoka.Config.default_model/0 or in your agent;
  • provider credentials that ReqLLM can use;
  • network access to the provider;
  • enough provider quota for at least two model calls.

A tool loop normally needs one model call to request the operation and one model call to produce the final answer.

Configure Credentials

The Jidoka source repository disables ReqLLM's automatic .env loading for package commands. Export a provider key in the shell that runs this live check.

In a host application, ReqLLM loads .env from the current working directory by default. To make the host application or deployment platform own credential loading, set this configuration:

# config/runtime.exs
import Config

config :req_llm, load_dotenv: false

For example:

export OPENAI_API_KEY=...
# or
export ANTHROPIC_API_KEY=...

Do not put provider keys in agent definitions, guide examples, test fixtures, snapshots, or trace metadata.

Inspect Before The Live Call

Use preflight to check the exact prompt and operations without network access:

{:ok, preflight} =
  Jidoka.preflight(
    MyApp.TimeAgent,
    "What time is it in Chicago? Use local_time."
  )

Enum.map(preflight.prompt.operations, & &1.name)
#=> ["local_time"]

Check the model reference, tool name, tool description, parameters schema, and control policy before you spend provider quota.

Run The Packaged Live Test

The standard test configuration excludes tests tagged with :live. Run the packaged live test explicitly:

mix test --include live test/jidoka/live_req_llm_test.exs

The test skips when neither OPENAI_API_KEY nor ANTHROPIC_API_KEY is set. When credentials are available, the test verifies that:

  • the Spark DSL compiles to a Jido-backed agent module;
  • ReqLLM makes a real model call;
  • the model requests the local_time operation;
  • the operation runs through the Jido action adapter;
  • the operation result is added to semantic agent state;
  • the model uses the operation result in its final answer;
  • the effect journal records the complete loop.

The tool returns a canary value. The final response must contain that value. This prevents a model from passing the test with a plausible answer that did not use the operation result.

Run An Application Agent

After preflight succeeds, run the same agent through the public facade:

case Jidoka.turn(MyApp.TimeAgent, "What time is it in Chicago? Use local_time.") do
  {:ok, result} ->
    IO.puts(result.content)
    IO.inspect(result.usage, label: "usage")
    IO.inspect(result.journal.results, label: "effect results")

  {:hibernate, snapshot} ->
    IO.inspect(snapshot.metadata, label: "pending review")

  {:error, reason} ->
    IO.puts(Jidoka.format_error(reason))
end

Use Jidoka.turn/3 for the live check because it returns the journal, events, usage, and snapshot data. Use Jidoka.chat/3 when application code needs only the final text.

Decision Protocol

Jidoka.Adapter.ReqLLM currently uses a constrained JSON decision protocol. The system prompt tells the model to return one of these shapes.

A final answer has this shape:

{"type":"final","content":"answer"}

A single operation request has this shape:

{"type":"operation","name":"local_time","arguments":{"city":"Chicago"}}

A parallel operation request has this shape:

{
  "type": "operations",
  "operations": [
    {"name": "lookup_customer", "arguments": {"id": "C100"}},
    {"name": "lookup_order", "arguments": {"id": "O200"}}
  ]
}

Jidoka.Adapter.ReqLLM.Decision parses the provider text into Jidoka.Effect.LLMDecision. The turn runner then plans effect intents. The effect interpreter executes them through injected runtime capabilities.

Treat invalid JSON and unknown decision types as model-output errors. Do not run an operation directly from unparsed provider text.

Production Checks

Before you enable live traffic, verify these items:

CheckReason
Set explicit model and generation defaultsProvider defaults can change behavior and cost.
Set turn and provider timeoutsA provider stall must not block work without a bound.
Apply operation controlsSide-effecting work needs explicit policy.
Use safe idempotency valuesResume and retry must not repeat unsafe work.
Persist sessions and snapshotsIn-memory state does not survive process or node loss.
Configure trace redactionPrompts, tool arguments, and results can contain sensitive data.
Record usage and provider errorsOperators need cost and failure evidence.
Limit concurrencyParallel model or operation calls can consume quota quickly.

See Configuration, Idempotency And Safety, and Tracing And Events for these controls.

Test Strategy

Use three levels of tests:

  1. Unit tests inject fake llm: and operations: capabilities. They must be deterministic and fast.
  2. Integration tests verify the DSL, operation adapters, controls, journal, and resume behavior without a provider.
  3. A small opt-in live test verifies credentials, provider access, ReqLLM, and the complete tool loop.

Do not make the normal test suite depend on provider availability. A provider failure must not hide a deterministic Jidoka regression.

Troubleshooting

SymptomCheck
The test skipsExport OPENAI_API_KEY or ANTHROPIC_API_KEY in the same shell.
Credential errorCheck the key name, provider account, and ReqLLM model provider.
Model returns invalid JSONUse a model that follows structured instructions and keep generation settings conservative.
Model returns a final answer without a toolCheck preflight output, tool descriptions, and agent instructions.
Operation is missingCheck that the tool is present in preflight.prompt.operations.
Operation runs but the final answer ignores itAdd a deterministic canary to the test operation result.
Turn hibernatesA control requires review. Inspect pending review data and resume with a valid response.
Request times outCheck turn timeout, ReqLLM timeout, network access, and provider status.
Live test is flakyKeep it opt-in and diagnose provider behavior separately from deterministic tests.

Reference