AutoCouncil: Auto-routing reference

CouncilEx.AutoCouncil is an opt-in routing layer for callers that don't know up-front which council fits a given prompt. It is purely additive: existing static (use CouncilEx) and dynamic (%DynamicCouncil{}) councils are unaffected.

Mental model. From the runner's perspective, an AutoCouncil is a council. Pass it to CouncilEx.run/3 exactly like any other council. Internally, a strategy picks an underlying council from a catalog and the runner executes that. The picked council surfaces in result.metadata.auto.

1. Quick start

auto =
  CouncilEx.AutoCouncil.new(
    name:     "general-routing",
    strategy: :rules,
    catalog:  [
      %{id: "seo",  council: MyApp.Councils.SEO,        match: ~r/seo|sitemap/i, desc: "SEO audit"},
      %{id: "code", council: MyApp.Councils.CodeReview, match: ~r/code|PR/i,     desc: "Code review"}
    ],
    on_no_match: {:fallback, MyApp.Councils.GeneralPurpose}
  )

{:ok, result} = CouncilEx.run(auto, %{question: "audit my SEO"})

result.metadata.auto
# => %{strategy: :rules, kind: :static, catalog_id: "seo", reason: "matched ~r/seo|sitemap/i", latency_ms: 1, score: nil}

2. Struct shape

%CouncilEx.AutoCouncil{
  strategy:       :rules | :cascade | :embedding | :llm_classify | :llm_build | {module(), keyword()},
  catalog:        [catalog_entry()] | {:registry, :council} | nil,
  on_no_match:    :error | {:fallback, council_or_id} | nil,
  name:           String.t() | nil,
  options:        keyword(),    # strategy-specific
  provider_check: boolean()     # default false; see Provider awareness below
}

Catalog entry:

%{
  required(:council) => module() | %CouncilEx.DynamicCouncil{},
  optional(:id)      => String.t(),
  optional(:desc)    => String.t(),                  # used by LLM strategies
  optional(:match)   => Regex.t() | (binary() -> boolean()),  # used by Rules
  optional(:vector)  => [float()],                   # used by Embedding
  optional(:tags)    => [atom()]
}

Only :council is required. Strategies ignore fields they don't need, so the same catalog can serve multiple strategies side-by-side.

3. Built-in strategies

3.1 :rules

First catalog entry whose :match field matches the prompt wins. :match may be a Regex.t() or a 1-arity boolean function over the prompt string. Entries without :match are never picked by this strategy.

Cost: zero. No LLM calls. Pure pattern matching.

AutoCouncil.new(
  strategy: :rules,
  catalog:  [
    %{id: "seo",     council: MyApp.Councils.SEO,        match: ~r/seo/i},
    %{id: "support", council: MyApp.Councils.Support,    match: fn t -> String.contains?(t, "ticket") end}
  ]
)

Returns {:error, :no_match} if nothing fits. See :on_no_match for fallback control.

3.2 :cascade

Try a chain of strategies in order; first success wins. Chain comes from the parent's :options:

AutoCouncil.new(
  strategy: :cascade,
  catalog:  catalog,
  options:  [chain: [:rules, :embedding, :llm_classify]]
)

Per-step outcomes emit a [:council_ex, :auto_council, :cascade_step] telemetry event so operators can see which steps earn their keep.

3.3 :embedding: stub

Planned: cosine similarity over precomputed catalog vectors. Today returns {:error, :not_implemented}. Design notes:

• Catalog entries provide a :vector field (precomputed at registration time).
• :options carries :embedder (1-arity fn binary -> [float]) and :threshold.
• Open: vector storage location, re-embed-on-edit signal, similarity function.

See module CouncilEx.AutoCouncil.Strategies.Embedding.

3.4 :llm_classify: stub

Planned: a cheap LLM picks one catalog id given the prompt and the catalog descriptions. Today returns {:error, :not_implemented}. Design notes:

• Catalog entries must carry :id and :desc.
• :options carries :profile (registered profile name), :max_tokens, optional :system_prompt override.
• Open: structured output enforcement, retry-on-parse-failure policy, per-route budget cap.

See module CouncilEx.AutoCouncil.Strategies.LLMClassify.

3.5 :llm_build: stub

Planned: an LLM synthesizes a fresh %DynamicCouncil{} for the prompt (members, system prompts, rounds, chair). The runner then executes that synthesized council. Decision returns as {:built, dc} so callers can see the plan was non-deterministic.

AutoCouncil.new(
  strategy: :llm_build,
  options:  [
    profile:        "openai_balanced",
    member_palette: [...],
    max_members:    5,
    allowed_rounds: [:independent_analysis, :peer_critique, :consensus_vote]
  ]
)

Hard constraints to enforce inside the strategy: bounded member count, restricted round set, mandatory DynamicCouncil.validate/1 pass, hard planner timeout. See module CouncilEx.AutoCouncil.Strategies.LLMBuild.

3.6 Custom strategy

Any module implementing CouncilEx.AutoCouncil.Strategy:

defmodule MyApp.AutoCouncil.UserPreference do
  @behaviour CouncilEx.AutoCouncil.Strategy

  @impl true
  def resolve(prompt, %CouncilEx.AutoCouncil{} = auto) do
    # Pull catalog if you need it; the helper handles inline + registry forms.
    entries = CouncilEx.AutoCouncil.Resolver.catalog_entries(auto)
    user    = Process.get(:current_user)
    # ...your decision logic.
    {:ok, {:static, MyApp.Councils.Default}, "user default"}
  end
end

AutoCouncil.new(strategy: {MyApp.AutoCouncil.UserPreference, []})

Return value:

{:ok, decision, reason :: String.t() | nil}
{:ok, decision, reason, score :: float() | nil, catalog_id :: String.t() | nil}
{:error, term()}

Where decision is:

• {:static, module()}: picked a use CouncilEx module.
• {:dynamic, %DynamicCouncil{}}: picked a registered/inline data council.
• {:built, %DynamicCouncil{}}: synthesized fresh; non-deterministic.

The 3-tuple form is sugar; the resolver normalizes it to the 5-tuple internally.

4. Catalog sources

# Inline: explicit, easy to test, lives with the calling code.
AutoCouncil.new(strategy: :rules, catalog: [%{id: "seo", ...}])

# Registry-backed: shared, hot-reloadable, UI-editable.
AutoCouncil.new(strategy: :rules, catalog: {:registry, :council})

Register entries in config:

config :council_ex, :registry,
  councils: %{
    "seo" => %{council: MyApp.Councils.SEO, match: ~r/seo/i, desc: "SEO audit"}
  }

Or at runtime:

:ok = CouncilEx.Registry.register_council("editorial", %{
  council: editorial_dynamic_council,
  match:   ~r/editorial|copy/i,
  desc:    "Editorial review"
})

CouncilEx.Registry.register_council/2 requires the :council key: mistakes raise FunctionClauseError at registration time, not at route time.

4b. Provider awareness

A catalog entry pointing at a council that uses an unconfigured provider will fail at execution time. Set provider_check: true on the AutoCouncil to drop those entries before strategies run.

AutoCouncil.new(
  strategy: :rules,
  catalog:  [...],
  provider_check: true
)

How providers are discovered

The sentinel :__sub_council__ is filtered out and replaced with the target's real providers: a parent's :providers set covers the whole nested tree.

Caching

Provider sets are cached on the catalog entry under :providers:

• CouncilEx.Registry.register_council/2 computes and stashes :providers at registration time. Pass an explicit MapSet (or nil) to override the auto-computed value.
• AutoCouncil.new/1 warms the cache for inline catalogs only when provider_check: true is set, so the default path stays free.
• A cache miss (registry-backed entry registered before provider_check existed, or explicit :providers => nil) falls back to a live Providers.eligible?/1 call.

Invalidation

Re-register (or rebuild the inline catalog) when a council's provider mix changes. There is no automatic detection; the cache is intentionally simple. Static councils are effectively free; dynamic councils pay one to_spec/1 walk per registration.

Telemetry

[:council_ex, :auto_council, :catalog_filtered]
  measurements: %{filtered_out_count: int, kept_count: int}
  metadata:     %{auto_name, configured: [atom], dropped_ids: [String.t()]}

Fires only when at least one entry is filtered out. Helpful for spotting load-order races (provider config arrives after catalog registration) and missing API keys in production.

Failure-mode philosophy

Providers.used/1 returns {:error, term()} when introspection fails (e.g. profile module not loaded). The resolver treats {:error, _} as "keep the entry": better to attempt the run and surface a clear provider-config error than to silently drop a council due to a load-order race. :filtered_out_count only counts entries that were successfully introspected and found ineligible.

Public helpers

{:ok, %MapSet{}} = CouncilEx.AutoCouncil.Providers.used(MyCouncil)
{:ok, %MapSet{}} = CouncilEx.AutoCouncil.Providers.used(%DynamicCouncil{} = dc)
%MapSet{}        = CouncilEx.AutoCouncil.Providers.configured()
true_or_false    = CouncilEx.AutoCouncil.Providers.eligible?(MyCouncil)

from_spec/1 is the underlying spec-walking function. Useful when implementing custom strategies that need provider info.

5. on_no_match policy

Controls what happens when the strategy can't pick anything.

The fallback decision is recorded in result.metadata.auto with fallback?: true and reason: "on_no_match fallback".

6. Result metadata

result.metadata.auto
# => %{
#   strategy:   :rules | :cascade | :embedding | :llm_classify | :llm_build | atom_of_custom_module,
#   kind:       :static | :dynamic | :built,
#   catalog_id: "seo" | nil,
#   reason:     "matched ~r/seo/i" | nil,
#   score:      0.83 | nil,
#   latency_ms: 2,
#   fallback?:  true   # only present when on_no_match fired
# }

kind semantics:

• :static: a use CouncilEx module ran.
• :dynamic: a pre-existing %DynamicCouncil{} (from catalog or fallback) ran. Reproducible.
• :built: an LLM synthesized the council fresh. Non-deterministic; running the same prompt twice will pick a different shape unless you opt into caching.

7. CouncilEx.auto/1,2 shortcut

For apps with a single default AutoCouncil, the call site can skip constructing the struct:

config :council_ex, :auto,
  strategy:    :cascade,
  catalog:     {:registry, :council},
  on_no_match: :error,
  options:     [chain: [:rules, :llm_classify]]

CouncilEx.auto(%{question: "..."})                      # uses configured default
CouncilEx.auto(%{question: "..."}, strategy: :rules)    # per-call override

Per-call opts are merged on top of the configured default. Struct opts (:strategy, :catalog, :on_no_match, :name, :options, :provider_check) build the router; everything else is forwarded to run/3, so you can pass :verbose, :verbose_io, or :await_timeout in the same keyword list:

CouncilEx.auto(%{question: "audit my SEO"}, verbose: true)

With no config and no overrides, CouncilEx.auto/1 returns {:error, :no_auto_config}.

Returns the same {:ok, %Result{}} shape as CouncilEx.run/3; inspect result.metadata.auto to see what was picked.

8. Telemetry

[:council_ex, :auto_council, :decision]          # one per resolve
  measurements: %{latency_ms: integer}
  metadata:     %{auto_name, strategy, kind, catalog_id, error}
                # `error` (string) present only on the failure path; absent on success

[:council_ex, :auto_council, :cascade_step]      # one per step inside :cascade
  measurements: %{latency_ms: integer}
  metadata:     %{auto_name, strategy, status: :ok | :error, reason}
                # `reason` is nil on :ok steps

[:council_ex, :auto_council, :catalog_filtered]  # only when provider_check drops ≥1 entry
  measurements: %{filtered_out_count: int, kept_count: int}
  metadata:     %{auto_name, configured: [atom], dropped_ids: [String.t()]}

:decision fires once per resolve/2 call, including via the CouncilEx.run/3 integration. :cascade_step fires once per attempted step inside a :cascade. :catalog_filtered fires only when at least one entry is dropped; see Provider awareness above for handler guidance.

9. Composability

Because AutoCouncil is a council from the runner's perspective:

• An AutoCouncil may resolve to another AutoCouncil (cascading routing layers).
• An AutoCouncil can be a sub-council member of a %DynamicCouncil{}.
• A built %DynamicCouncil{} can itself contain sub-councils.

No special-case wiring; the standard runner handles all of these.

10. Public API

11. Deferred / open questions

These are intentionally out of scope for the initial slice. Notes live inside the relevant moduledocs (CouncilEx.AutoCouncil.Strategies.{Embedding,LLMClassify,LLMBuild}).

• Cache: prompt → decision cache for :llm_* strategies. Open: key shape (exact vs semantic vs embedding bucket), storage (ETS / Cachex / user-supplied), invalidation on catalog edits, opt-in semantics for :built (since caching turns a non-deterministic feature into a deterministic one).
• Async API: CouncilEx.start/3 does not yet have an %AutoCouncil{} clause. The sync run/3 path is supported. Wiring async needs threading the routing meta through the RunServer so it lands on the Result at completion.
• Vectors: where embedding vectors live (alongside the registry entry vs a parallel store), staleness signal on description edits.
• :llm_build schema enforcement: structured output vs prompt + post-validate. Telemetry on synthesized plans (member count, prompt lengths). Optional promotion of synthesized plans into the registry (would cross from :built into :dynamic on subsequent hits).

12. See also

• README.md: Auto-routing section
• DYNAMIC_COUNCILS.md: %DynamicCouncil{} reference; AutoCouncil's :built decisions return data councils.
• COMPOSITION.md: sub-councils + adaptive routers; orthogonal to AutoCouncil but composes with it.
• COUNCILS.md: pre-built static topologies (Specialist, Consensus, Tournament, WeightedConsensus, JuryWithRetry, …), common registrants in an AutoCouncil catalog.
• RELATED_WORK.md: map of multi-agent LLM papers and projects (Council Mode, MAD, Adjudicator, karpathy/llm-council, etc.) onto CouncilEx capabilities.