Dynamic Councils: reference

Reference for building, validating, persisting, and running %CouncilEx.DynamicCouncil{} data-form councils. The README has a quickstart tour; this document covers the contracts a production consumer needs.

Source-of-truth modules:

• lib/council_ex/dynamic_council.ex
• lib/council_ex/dynamic_member.ex
• lib/council_ex/dynamic_round.ex
• lib/council_ex/registry.ex
• lib/council_ex/member_spec.ex
• lib/council_ex/runner/run_server.ex

1. Overview

Module-form councils (use CouncilEx) are compile-time, tied to a module, and best when the topology is part of the application source. Routers, members, chair, rounds are all expressible as Elixir code, including inline closures that capture lexical state.

Dynamic form (%DynamicCouncil{}) is a plain data structure: build it at runtime, persist as JSON, restore, edit field-by-field, run. Designed for UI-driven workflow builders (React Flow, etc.) and DB-backed council libraries where the user owns the topology, not the developer. Both forms run through the same Runner.RoundExec path via CouncilEx.MemberSpec.

2. Quick start

alias CouncilEx.DynamicCouncil

council =
  DynamicCouncil.new("seo-1", name: "SEO audit")
  |> DynamicCouncil.set_default_profile("openai_mini")
  |> DynamicCouncil.add_member(%{id: "seo", system_prompt: "Audit SEO."})
  |> DynamicCouncil.add_member(%{id: "content", system_prompt: "Audit content."})
  |> DynamicCouncil.add_round(:independent_analysis)
  |> DynamicCouncil.set_chair(%{id: "synth", system_prompt: "Synthesize."})

:ok = DynamicCouncil.validate!(council)
{:ok, result} = CouncilEx.run(council, %{question: "..."})

Full version (including JSON round-trip, React-Flow export, runtime profile registration): examples/dynamic_council_example.exs.

3. The Registry

CouncilEx.Registry is the lookup table for everything dynamic councils reference by string name.

3.1 Kinds

3.2 Storage tiers

Two tiers, both readable through the same lookup API. Runtime wins on conflict.

# Boot-time, in config/config.exs:
config :council_ex, :registry,
  profiles: %{"openai_mini" => CouncilEx.Profiles.OpenAIMini},
  tools:    %{"calculator"  => MyApp.Tools.Calculator}

# Runtime, anywhere:
:ok = CouncilEx.Registry.register_sub_council("seo_pipeline", inner_council)
:ok = CouncilEx.Registry.register_input_mapper("topic_to_brief",
        &MyApp.Mappers.topic_to_brief/1)

Config keys are plural (:profiles, :tools, :schemas, :routers, :rounds, :sub_councils, :input_mappers, :councils); the lookup API uses singular kinds. The :council kind backs the AutoCouncil catalog (see AUTO_COUNCILS.md).

Design note: no web/HTTP in core. Tools are Elixir modules registered by string name; UI users pick from the registered set. Adding a new tool behaviour is code + registration. Adding a new tool configuration is data. The runtime never performs HTTP calls on behalf of the council graph itself — only member adapters do, inside their own supervised tasks.

3.3 Lookup contract

CouncilEx.Registry.lookup(:profile, "openai_mini")    # => module() | nil
CouncilEx.Registry.lookup!(:profile, "openai_mini")   # raises with known names
CouncilEx.Registry.list(:profile)                     # => sorted [String.t()]
CouncilEx.Registry.all(:profile)                      # => merged map
CouncilEx.Registry.unregister(:tool, "calculator")    # runtime entry only
CouncilEx.Registry.reset_runtime()                    # wipe all runtime entries

4. Sub-councils in dynamic form

A %DynamicMember{} can wrap an entire inner council.

4.1 Three ref shapes

# (a) Inline nested struct: ad-hoc composition.
inner = DynamicCouncil.new("inner") |> ...

DynamicCouncil.add_member(outer, %{id: "marketing", sub_council: inner})

# (b) Registered name string: UI / persistence flows.
:ok = CouncilEx.Registry.register_sub_council("seo_pipeline", inner)
DynamicCouncil.add_member(outer, %{id: "marketing", sub_council: "seo_pipeline"})

# (c) Module atom: reuse an existing module-form council.
DynamicCouncil.add_member(outer, %{id: "marketing", sub_council: MyApp.SEOCouncil})

When :sub_council is set, :profile, :tools, :output_schema are ignored: the member's output is the wrapped sub-run result. Members with :sub_council do not require :system_prompt.

4.2 :input_mapper

Projects the parent council's input before the inner council sees it.

# Three accepted shapes:
input_mapper: nil                                 # identity (default)
input_mapper: "topic_to_brief"                    # registered name (preferred)
input_mapper: &MyApp.Mappers.topic_to_brief/1     # remote capture

Closure constraint. Anonymous fn x -> ... end works at runtime but does not JSON-round-trip: to_json/1 emits %{"capture" => "anonymous"} and from_json/1 decodes it as an unresolved sentinel that validate/1 flags. Remote captures (&Mod.fun/1) emit %{"capture" => "&Mod.fun/1"} informationally but also do not round-trip executably: only registered-name strings survive the round-trip cleanly. Use the registry for any persisted council.

4.3 Cancel cascade

Calling CouncilEx.cancel(pid) (or cancel(run_id)) on the outer run cascades to all active sub-runs first (RunServer.handle_cast(:cancel, _) walks state.sub_run_ids and cancels each before killing the parent task). Behaviour is identical for module-form sub-councils (SubCouncilAdapter) and dynamic sub-council members.

4.4 Demo

examples/dynamic_sub_council_example.exs walks all three ref shapes plus an :input_mapper example, using the Mock provider (no API key required).

4.5 Hybrid form: static outer, dynamic inner

The reverse direction also works: a static (use CouncilEx) council can host an inline %DynamicCouncil{} as a sub-council member.

defmodule MyApp.OuterStatic do
  use CouncilEx

  member :researcher do
    provider :mock
    model "mock"
    system_prompt "research"
  end

  # Inline expression: the pipeline evaluates at compile time of
  # MyApp.OuterStatic and the resulting struct is captured into the
  # SubCouncilAdapter shim.
  member :seo,
    council:
      CouncilEx.DynamicCouncil.new("seo")
      |> CouncilEx.DynamicCouncil.add_member(%{
           id: "keyword",
           system_prompt: "extract keywords",
           profile_overrides: [provider: :mock, model: "mock"]
         })
      |> CouncilEx.DynamicCouncil.add_round(:independent_analysis)

  round :independent_analysis
end

This means forms mix in either direction across the sub-council boundary:

Dynamic outer hosting a static sub-council:

outer =
  CouncilEx.DynamicCouncil.new("outer")
  |> CouncilEx.DynamicCouncil.set_default_profile("openrouter_default")
  |> CouncilEx.DynamicCouncil.add_member(%{id: "strategist", system_prompt: "..."})
  # Reference an existing battle-tested static council module:
  |> CouncilEx.DynamicCouncil.add_member(%{
       id: "marketing",
       sub_council: MyApp.MarketingCouncil
     })
  |> CouncilEx.DynamicCouncil.add_round(:independent_analysis)

Within a single dynamic council, members can mix freely: regular members alongside sub-council members pointing at static modules, nested dynamic structs, or registered names, all in the same add_member chain.

Use cases:

• Static codebase, user-authored extensions. Ship a curated set of static councils; let users compose them at runtime in dynamic councils built from a UI.
• Dynamic per-tenant flows reusing static building blocks. Tenants configure the outer flow visually. Performance-critical or security-sensitive sub-flows stay in compile-checked static councils.
• Migration path. Convert one round or one branch from static to dynamic without rewriting the whole council.

End-to-end coverage: test/council_ex/hybrid_form_test.exs.

The dual-form pattern (§8) runs the same council in either form. Hybrid form runs different forms together in one council. Different feature, often confused.

4.6 Council-level tools (RAG entry point)

Tools default to per-member, but %DynamicCouncil{} carries a council-level :tools field as well. Anything attached there is merged into every regular member's resolved tool list, so the whole council shares a toolset (a retrieval tool the whole council debates against, a calculator, a notes scratchpad).

DynamicCouncil.new("c-rag")
|> DynamicCouncil.set_default_profile("openrouter_default")
|> DynamicCouncil.add_council_tool(MyApp.Tools.PolicyDocs)        # shared
|> DynamicCouncil.add_member(%{
     id: "legal",
     tools: [MyApp.Tools.LegalPrecedents],                        # specialist
     system_prompt: "Flag liability + regulatory exposure."
   })
|> DynamicCouncil.add_round(:independent_analysis)

Builders: add_council_tool/2, set_council_tools/2. Refs are the same shape as member tools: registered name (string) or module atom.

Merge rules:

• Council tools are prepended to each member's list, then deduped at the module level (after registry resolution). A council ref "calc" and a member ref Calc collapse to one tool.
• The chair receives council tools the same as members.
• Sub-council members are excluded. The inner run owns its own toolset; attach tools inside the sub-council if needed.
• JSON ser/de round-trips council tools the same way as member tools (string refs stay strings; module atoms are emitted as %{"module" => "..."} markers).

Validation: unknown council-level refs surface at ["tools", N] with code :unknown (alongside the existing ["members", N, "tools", K] member-level path).

This is the phase-1 RAG surface. See docs/RAG.md for the end-to-end pattern, the CouncilEx.Tools.InMemoryDocs helper, and the trade-offs vs the deferred pre-injection retriever.

5. Validation

:ok | {:error, [%{path: [...], code: atom(), message: String.t()}]}
  = DynamicCouncil.validate(council)

Errors are plain maps with no Elixir struct rendering, so they paste straight into a JSON API response or React form.

5.1 Common error codes

5.2 Always validate before persisting and before running

to_spec/1: the function CouncilEx.run/3 calls to lower a %DynamicCouncil{} to a runner spec: does not auto-validate. It will happily raise deep inside profile/tool/schema resolution if a ref is missing. Validate at the boundary:

with :ok <- DynamicCouncil.validate(council) do
  # safe to persist + run
end

validate!/1 raises ArgumentError with the error list inspected.

start/3 now gates on validation automatically. Invalid councils return {:error, {:invalid_council, errs}} from start/3 itself — no RunServer spawned, no PubSub broadcast. The same checks run for module-form councils via CouncilEx.validate/1. You can still call DynamicCouncil.validate/1 directly when you need the structured errors before dispatch (form validation, persistence gate, etc.). start/3 just removes the "forgot to validate, ate a KeyError deep in a Task" failure mode.

6. JSON serialisation

DynamicCouncil.to_json(council, opts \\ [])  # {:ok, json} | {:error, term}
DynamicCouncil.to_json!(council, opts \\ [])
DynamicCouncil.from_json(json)               # {:ok, council} | {:error, term}
DynamicCouncil.from_json!(json)              # raises on invalid JSON

Underneath: to_map/1 + from_map/1 give you the same shape as plain maps, useful when the wrapping JSON encoding is handled elsewhere.

6.1 Schema versioning

Every council carries a version: 1 field (@current_version). from_json/1 calls migrate(version, attrs) between decoded form and struct construction. Versions newer than the current build raise ArgumentError:

unsupported DynamicCouncil schema version <N>; this build understands up to v1

Add migration clauses to migrate/2 as the schema evolves.

6.2 String-keyed JSON → atom struct

atomize/1 calls String.to_existing_atom/1 (not String.to_atom/1) on every key, guarded by try/rescue that re-raises with unknown <Module> schema field: .... Legitimate field names are interned at compile time via defstruct, so they always exist as atoms when JSON arrives. This blocks atom-table exhaustion from untrusted JSON input. The :metadata field is a free-form map and its internal keys are NOT atomized.

6.3 Round-trip stability

Rule of thumb: any council that touches persistent storage should reference closures by registered name only.

7. Polymorphic dispatch

CouncilEx.run/3 and CouncilEx.start/3 dispatch on the council argument:

CouncilEx.run(MyApp.MyCouncil, input, opts)            # module-form
CouncilEx.run(%DynamicCouncil{} = council, input, opts) # dynamic form

For dynamic councils, Result.council and PubSub event metadata carry the string "dynamic:" <> council.id.

7.1 Migration from pre-release

run_dynamic/3 and start_run_dynamic/3 were removed with no deprecation cycle (pre-release API). Migration is a one-liner:

# before
CouncilEx.run_dynamic(council, input)

# after
CouncilEx.run(council, input)

8. Dual-form pattern

Several real-provider examples use a COUNCIL_FORM=static|dynamic environment switch so the same example file demonstrates both shapes:

council =
  case System.get_env("COUNCIL_FORM", "static") do
    "static"  -> MyApp.MyCouncil
    "dynamic" -> build_dynamic_council()
  end

{:ok, result} = CouncilEx.run(council, input)

When to use the dual form: A/B verification that a topology behaves identically in both shapes, or as a side-by-side teaching reference.

mix run examples/parallel_panel_example.exs                       # static (default)
COUNCIL_FORM=dynamic mix run examples/parallel_panel_example.exs  # dynamic

Examples that support COUNCIL_FORM: parallel_panel, tool_calling, anthropic_streaming, anthropic_structured_output, gemini, ollama, streaming, phoenix_pubsub, router, creative_judge, debate, multi_model_panel, openrouter (16+ files).

9. Telemetry

RunServer emits run/round events on the modern async path so the full event surface fires for both run/3 and start/3.

| [:council_ex, :tool, :execute, :*] | (existing) | includes path: :complete | :stream |

Per-event cost: ~3µs per attached handler. See bench/telemetry_overhead.exs for the methodology.

10. Prebuilt Council wrappers (new_dynamic/1)

Four of the prebuilt councils expose a dynamic-form constructor that returns a %DynamicCouncil{} instead of generating a module:

Councils.JuryWithRetry is static-only. The iterate+retry shape (ETS-backed convergence callback registry) is non-trivial to round-trip through JSON serialization. Use Councils.JuryWithRetry.new/1 directly or compose Rounds.Iterate(:independent_analysis, until: …) by hand in a %DynamicCouncil{} if you need the data form.

Members and chair accept the same shapes the underlying DynamicCouncil builders accept (struct, map, or keyword list).

council =
  CouncilEx.Councils.Specialist.new_dynamic(
    id: "seo-audit",
    members: [
      %{id: "seo",  role: "SEO Expert",          system_prompt: "Audit SEO issues."},
      %{id: "tech", role: "Technical Architect",  system_prompt: "Audit tech stack."}
    ],
    chair: %{id: "synth", role: "Synthesizer", system_prompt: "Top three actions."},
    default_profile: "openai_mini"
  )

{:ok, result} = CouncilEx.run(council, %{topic: "..."})

11. Operational concerns

• Validate twice. Once before persisting (validate/1 → {:error, errs} straight to UI). Once before running (defence in depth: registry contents may have changed since persist).
• Profile registry must be populated before to_spec/1 runs. CouncilEx.run/3 calls to_spec/1 synchronously; missing profile entries raise ArgumentError with the list of registered profiles.
• Cancel cascade. CouncilEx.cancel(outer_run_id) cancels active sub-runs first, then kills the outer round task.
• PubSub topic format.
  • Outer run: "council_ex:run:#{run_id}".
  • Sub-runs ALSO relay each event onto a parent-prefixed topic: "council_ex:run:#{parent_run_id}:sub:#{member_id}". Subscribe to that topic to observe sub-run events without resolving the inner run_id separately.
• Telemetry overhead is sub-millisecond per event. Safe to attach in production.
• Inline JSON Schema (output_schema_inline:) bypasses Ecto. MemberSpec.output_schema/1 returns {:json_schema, map}; the Instructor provider returns the parsed map as-is on response.parsed (no struct).

12. What's NOT yet in dynamic form

• Chained / multi-step tool loops: every tool example is currently single-call. Chained tool-loop design notes are deferred.
• Adaptive routers as inline closures: router: accepts a registered router module name only in dynamic form; module-form councils additionally accept a 2-arity fn input, ctx -> [member_id] end.
• Custom rounds with closure-bearing opts: module-form round/2 detects function literals in opts and lifts them into a generated function; dynamic-form rounds carry a plain opts map only. Custom rounds with non-closure opts work fine via :round registry.
• Inline (do: block) members: module-form only; the equivalent in dynamic form is a %DynamicMember{} with :system_prompt set directly.