Unit testing councils

Copy Markdown View Source

How to write fast, deterministic tests for councils without spending API quota. CouncilEx ships a Mock provider plus three test helpers that cover the common cases: end-to-end run assertions, per-round event assertions, and full event capture.

For manual / live-provider testing (real API keys, recipe per provider, no-key smoke tests), see TESTING.md. This document is about ExUnit-driven unit tests.

The Mock provider

CouncilEx.Providers.Mock is an in-memory provider that returns pre-scripted responses keyed by member id. It is only intended for tests and example fixtures: never for application code.

# Configure once in test/test_helper.exs
Application.put_env(:council_ex, :providers,
  mock: [dispatcher: CouncilEx.Providers.Mock]
)

Members route to it via provider: :mock, model: "mock".

Per-member scripting:

CouncilEx.Providers.Mock.script(:writer, content: "Lorem ipsum.")
CouncilEx.Providers.Mock.script(:writer, stream: ["Lo", "rem ", "ipsum."], chunk_delay_ms: 5)

Streaming scripts emit one chunk per element, with a configurable inter-chunk delay so streaming tests can exercise back-pressure paths.

CouncilEx.Test

Add import CouncilEx.Test in your test file. Three helpers:

script_council/2

Register a Mock script for every member of a council in one call. Three shapes for the second argument:

  • a string: every member receives the same content,
  • a map of member_id => string: per-member content,
  • a 2-arity fn fn member_id, member_module -> content_string end: called once per member.

script_council/2 recurses into sub-councils, so members of a nested council are scripted by the same call. Members that do not route to the Mock provider are silently skipped.

capture_events/2

Subscribe to a run's PubSub topic, drain every event until the terminal :run_completed / :run_failed event arrives or timeout elapses, then return the ordered event list.

events = capture_events(run_id, 5_000)
assert match?({:run_completed, _, _}, List.last(events))

Subscription happens inside the call, so events emitted before capture_events/2 returns (e.g., :run_started if the run has already started) are lost. To capture every event, subscribe directly via CouncilEx.PubSub.subscribe/1 before kicking off the run, then drain the mailbox yourself.

assert_round_completed/3

ExUnit assertion macro that blocks until a :round_completed event for the given round name arrives, or fails with the caller's file/line. Returns the matched %CouncilEx.RoundResult{} for further assertions.

CouncilEx.PubSub.subscribe("council_ex:run:#{run_id}")
rr = assert_round_completed :independent_analysis, run_id, 2_000
assert rr.member_results[:writer].response.content == "Lorem ipsum."

The caller must be subscribed to the run's topic before invoking the macro. Default timeout is 1000ms.

Full example

defmodule MyApp.CouncilTest do
  use ExUnit.Case, async: false
  import CouncilEx.Test

  test "runs end-to-end" do
    script_council(MyApp.MyCouncil, "constant reply")
    {:ok, pid} = CouncilEx.start(MyApp.MyCouncil, %{})
    run_id = CouncilEx.RunServer.run_id(pid)
    events = capture_events(run_id, 5_000)
    assert match?({:run_completed, _, _}, List.last(events))
  end

  test "round emits expected event" do
    script_council(MyApp.MyCouncil, %{a: "ay", b: "bee"})
    {:ok, pid} = CouncilEx.start(MyApp.MyCouncil, %{})
    run_id = CouncilEx.RunServer.run_id(pid)
    CouncilEx.PubSub.subscribe("council_ex:run:#{run_id}")

    rr = assert_round_completed :independent_analysis, run_id, 2_000
    assert rr.member_results[:a].response.content == "ay"
  end
end

State inspection

For introspection during a test (or in production, e.g. from a LiveView dashboard):

# Curated summary of a single run:
{:ok, summary} = CouncilEx.RunServer.state(run_id)
# %{run_id: ..., status: :running, current_round: %{name: ..., idx: 0}, ...}

# All currently active or recently-completed runs:
runs = CouncilEx.list_active_runs()

state/1 returns a stable curated map. The internal RunState shape may change across versions; summary keys are public. list_active_runs/0 runs the per-process state queries concurrently with a 1s timeout per run.

Tips

  • Prefer start + assert_round_completed over run/3 when asserting on intermediate round shape: the sync run/3 only returns the final %Result{}.
  • Subscribe with CouncilEx.start(council, input, subscribe: true) to eliminate the subscribe-after-broadcast race; the caller is subscribed before the run process boots.
  • Use failure_mode: :continue on rounds whose tests assert partial failures, otherwise the run aborts at the first member error.
  • For tool tests, use the :tool_call_request and :tool_call_result PubSub events rather than reaching into MemberResult internals. Those events are part of the frozen CouncilEx.Events surface.