Contributor Testing

Jidoka's test suite is designed to stay deterministic by default, with a narrow opt-in path for live provider runs. Contributors who add a feature must also extend the deterministic surface (unit, runtime, golden, and integration tests) and only optionally extend the live tests. This guide documents the test layout, the fake-LLM and local-operation patterns, the golden-file contract, and the mix quality gates that every change must clear. It is written for people contributing to the jidoka package, not for application authors.

When To Use This

• Use this guide before writing a new test in test/. The folder structure and helper modules are not obvious from the file tree alone.
• Use this guide when adding a feature that should appear in golden coverage (any change to Agent.Spec, Turn.Plan, or the import path).
• Use this guide when changing default mix quality gates or when adding a new opt-in test category.
• Do not use this guide for application-level testing. Application authors should follow Testing And Evals.

Prerequisites

• Elixir ~> 1.18 and a checkout of the jidoka package.
• mix deps.get has been run.
• For the live opt-in path: one provider key in scope (OPENAI_API_KEY or ANTHROPIC_API_KEY).

mix deps.get
mix test

Quick Example

A deterministic test only needs a fake LLM function and (optionally) an injected operation capability. Both go through Jidoka.turn/3:

defmodule MyContributorTest do
  use ExUnit.Case, async: true

  import TestSupport

  defmodule TimeAgent do
    use Jidoka.Agent

    agent :contributor_time do
      model %{provider: :test, id: "deterministic"}
      instructions "Call local_time when asked for the time."
    end

    tools do
      local_operations do
        operation :local_time do
          description "Returns a fixed time for a city."
          handler fn %{"city" => city} -> {:ok, %{city: city, time: "09:30"}} end
        end
      end
    end
  end

  test "returns the canned answer with no provider key" do
    llm = operation_then_final_llm("local_time", %{"city" => "Chicago"}, "Chicago: 09:30")

    assert {:ok, %Jidoka.Turn.Result{content: "Chicago: 09:30"}} =
             Jidoka.turn(TimeAgent, "What time is it in Chicago?", llm: llm)
  end
end

No environment variable is required. The test runs in async: true because both capabilities are pure functions.

Concepts

Three ideas define the contributor test surface.

1. Deterministic by default, live by opt-in. test/test_helper.exs excludes the :live tag by default. Live tests must be tagged @moduletag :live and run with mix test --include live.
2. Two injection seams replace every external dependency. The llm: keyword option supplies a fake Jidoka.Runtime.Capabilities.llm_capability/0 function, and operations: supplies a fake Jidoka.Runtime.Capabilities.operation_capability/0 function. Together they make the runtime fully data-driven.
3. Golden tests pin the public projection. Any change to a struct that escapes the package boundary must update the matching golden expectation.

                  mix test
                     │
        ╭────────────┼─────────────╮
        ▼            ▼             ▼
   unit tests   runtime tests  golden tests
  (per module) (capabilities,    (DSL->spec,
   pure data)   interpreter,      import->spec)
                runner)
        │            │             │
        ╰────────────┼─────────────╯
                     ▼
            integration tests
            (test/integration/,
            scenario-shaped)
                     │
                     ▼
            mix test --include live
            (opt-in real provider)

How To

Step 1: Pick The Right Test Folder

The repo's test layout has four buckets. New tests go in the bucket that matches their scope:

Tests that need shared agents, actions, or controls go under test/support/integration/{agents,actions,controls}/. The test/support/integration/README.md file documents who lives there.

Step 2: Write A Fake LLM Capability

The shared helpers in test/support/test_support.ex cover the common shapes. The three building blocks are final_llm/2, operation_llm/2, and operation_then_final_llm/3:

def final_llm(content, opts \\ []) when is_binary(content) do
  result = Keyword.get(opts, :result)

  fn _intent, _journal, _ctx ->
    {:ok, %{type: :final, content: content, result: result}}
  end
end

def operation_llm(name, arguments \\ %{}) when is_binary(name) and is_map(arguments) do
  fn _intent, _journal, _ctx ->
    {:ok, %{type: :operation, name: name, arguments: arguments}}
  end
end

def operation_then_final_llm(name, arguments, content) do
  fn _intent, %Effect.Journal{} = journal, _ctx ->
    case count_results(journal, :llm) do
      0 -> {:ok, %{type: :operation, name: name, arguments: arguments}}
      _count -> {:ok, %{type: :final, content: content}}
    end
  end
end

The contract:

• The function takes (Effect.Intent.t(), Effect.Journal.t()) and returns {:ok, decision_map_or_struct} | {:error, term}.
• The decision can be a Jidoka.Effect.LLMDecision struct or a plain map matching the JSON decision shape.
• The function is called once per loop iteration. Use count_results/2 on the journal to branch by iteration number.

For multi-step loops, write a small reduction directly inline rather than a helper:

llm = fn _intent, %Effect.Journal{} = journal, _ctx ->
  case TestSupport.count_results(journal, :llm) do
    0 -> {:ok, %{type: :operation, name: "step_a", arguments: %{}}}
    1 -> {:ok, %{type: :operation, name: "step_b", arguments: %{}}}
    2 -> {:ok, %{type: :final, content: "done"}}
  end
end

Step 3: Write A Local Operation Capability

For tests that exercise tool calls, use Jidoka.Operation.Source.Local when the agent is defined through the DSL, or Jidoka.Runtime.LocalOperations.operations/1 when you want a bare capability function:

operations =
  Jidoka.Runtime.LocalOperations.operations(%{
    "local_time" => fn %{"city" => city} -> {:ok, %{city: city, time: "09:30"}} end
  })

Jidoka.turn(MyAgent, "input", llm: llm, operations: operations)

When the test agent declares operations through DSL, prefer Jidoka.Operation.Source.Local inside the DSL itself so the spec is self-contained and golden-testable.

The handler signatures:

Step 4: Author A Golden Test

Golden tests live in test/jidoka/golden/. The canonical shape is in test/jidoka/golden/dsl_to_spec_test.exs:

defmodule Jidoka.GoldenTest.Support.MinimalAgent do
  use Jidoka.Agent

  agent :golden_minimal_agent do
    model %{provider: :test, id: "golden-minimal-model"}
  end
end

defmodule Jidoka.Golden.DslToSpecTest do
  use ExUnit.Case, async: true

  alias Jidoka.GoldenTest.Support.MinimalAgent

  test "minimal DSL agent compiles to the expected Agent.Spec projection" do
    assert Jidoka.project(MinimalAgent.spec()) == %{
             id: "golden_minimal_agent",
             instructions: Jidoka.Agent.default_instructions(),
             model: "test:golden-minimal-model",
             generation: %{params: %{temperature: 0.0, max_tokens: 500},
                           provider_options: %{},
                           extra: %{}},
             context_schema?: false,
             result: nil,
             memory: nil,
             operations: [],
             controls: %{max_turns: nil, timeout_ms: nil,
                         inputs: [], outputs: [], operations: [],
                         metadata: %{}},
             runtime_defaults: %{},
             metadata: %{...}
           }
  end
end

Three rules for golden tests:

• Use == not =~. The whole point is to detect any drift.
• Co-locate the support modules in the same file. Each golden test module owns its fixtures so cross-file moves are obvious.
• Update the expected map in the same commit as the change. A green golden test after a struct change usually means you forgot to assert the new field.

Step 5: Author An Integration Test

Integration tests live in test/integration/ and mirror an author flow. They are still deterministic; they just exercise more than one module per test. Folder conventions:

Reuse the shared agents under test/support/integration/agents/ whenever a scenario fits one of them (MinimalChatAgent, AccountAgent, ControlledLookupAgent).

Step 6: Add A Live Test (Opt-In)

The live test pattern is documented in test/jidoka/live_req_llm_test.exs:

defmodule Jidoka.LiveReqLLMTest do
  use ExUnit.Case, async: false

  @moduletag :live
  @moduletag timeout: 120_000

  @live_enabled? not is_nil(System.get_env("OPENAI_API_KEY") || System.get_env("ANTHROPIC_API_KEY"))

  if @live_enabled? do
    # ... test bodies referencing real providers ...
  end
end

Three rules for live tests:

• Always tag @moduletag :live. The test/test_helper.exs excludes :live so default mix test stays fast.
• Guard with @live_enabled?. A live test without a key should compile but contain no test cases.
• Set a generous @moduletag timeout. Real providers vary; 120s is the current default.

Run live tests with mix test --include live.

Step 7: Clear mix quality Before You Push

The mix quality alias runs the gates defined in mix.exs:

quality: [
  "format --check-formatted",
  "compile --warnings-as-errors",
  "xref graph --format cycles --fail-above 0",
  "cmd env MIX_ENV=test mix test test/architecture/boundaries_test.exs",
  "credo",
  "dialyzer",
  "doctor --raise"
]

Each step is non-negotiable:

Run mix quality after every meaningful change. Do not push a branch that fails any of these.

Common Patterns

• Inject capabilities at the top of the test. A test that fakes the LLM inside a helper deep in the call chain is hard to follow. Keep the seam visible.
• Branch on the journal, not on test state. count_results(journal, :llm) is the canonical way to "do this on the first call, that on the second".
• Prefer Jidoka.project/1 over deep struct assertions. Asserting on a projection survives implementation churn that does not change the public shape.
• Use Jidoka.Trace.timeline/1 for event assertions. It shrinks event details to the stable trace shape.
• Group integration helpers in test/support/integration/. Per-file one-off agents accumulate noise.

Testing

The package itself is the test bed. Two cross-cutting commands matter:

# Fast, deterministic, default. Excludes :live.
mix test

# Include live tests. Requires a provider key.
mix test --include live

# Full quality bar.
mix quality

# Production coverage. The minimum is 75%.
mix test --cover
mix test.coverage

# Documentation warnings and alignment rules.
mix docs --warnings-as-errors
mix run scripts/check_docs.exs

Production coverage excludes the development-only Kino layer, test support modules, and complete example applications. These modules do not compile into the production application. DSL and verifier modules are counted because they implement production authoring and validation behavior.

For a single contributor change, the loop is usually:

mix test path/to/test_file.exs
mix format
mix quality

Troubleshooting

Reference

• test/support/test_support.ex - shared helpers: final_llm/2, operation_llm/2, operation_then_final_llm/3, timeline/1, event_index/2, operation_control_index/2, operation_capability_index/2.
• Jidoka.Runtime.LocalOperations - function-backed operation capability for tests.
• Jidoka.Operation.Source.Local - DSL surface that wraps LocalOperations for self-contained test agents.
• Jidoka.Runtime.Capabilities - typed bundle that the runner consumes; the llm_capability/0 and operation_capability/0 types are the test contract.
• Jidoka.Effect.Journal - the journal the fake LLM inspects to branch on iteration.
• Jidoka.project/1 - target of golden assertions.
• Jidoka.Trace - source of the timeline/1 helper used in event assertions.

Related Guides

• Testing And Evals - author-facing test patterns.
• Turn Runner And Effect Interpreter - how the loop the tests exercise actually runs.
• Runtime Capabilities Internals - the adapters the test capabilities mirror.
• Projection Internals - what golden tests are pinning.
• Troubleshooting - error reference for failures that surface during tests.