ExAthena.Loop.Inference (ExAthena v0.20.0)

Copy Markdown View Source

The single instrumented path for every provider call made inside a run.

ReAct's main turn, PlanAndSolve's planning turn, Reflexion's critique, the conclusion-distillation micro-call, and the compaction summary all route through call/3, which uniformly applies:

  • Request queue — the call holds a ExAthena.RequestQueue slot for its full lifetime (per-provider concurrency gating for scarce local GPU slots), with {:queue_wait, …} loop events emitted on real waits.
  • ChatParams hooks — hosts adjusting params per call (or halting) see every conversational inference, not just the main turn. Disable with chat_params: false for runtime utility calls whose fixed micro-budgets must not be reshaped by conversational hooks.
  • Telemetry — the [:ex_athena, :chat] span fires for every call, with GenAI-semconv metadata plus a :purpose tag (:turn, :planning, :reflection, :conclusion_distillation, :compaction_summary) so metering hosts can attribute cost.
  • Budget accounting — usage and cost fold into state.budget with full cost extraction (total_cost, or input_cost/output_cost summed on the fly), and a {:usage, usage} loop event is emitted.

Starvation policy (:starvation — required)

Adapters flag output-starved turns (response.starvation) so the kernel can retry the iteration once with an escalated max_tokens (see ExAthena.Loop, issue #194). That escalation raises the run's request-template completion cap ~4x for the REST of the run and re-runs the whole iteration — the right response when a full turn starves, and a disproportionate one when a fixed 256-token utility call does.

Every call site must therefore state its policy explicitly:

  • starvation: :surface — full, turn-shaped calls (main turn, planning). A starved response returns {:error, {:error_thinking_starved, info, folded_state}} so the kernel escalates; the folded state keeps the starved attempt's token burn on the budget across the retry.
  • starvation: :tolerate — internal micro-calls (critique, distillation, summary). A starved response returns {:ok, response, folded_state} like any other; the caller treats the blank text as a failed micro-call and falls back gracefully. The kernel's escalation must never fire for these — a mis-sized micro-call must not inflate the main turn's completion cap.

Summary

Types

option()

Functions

apply_chat_params(state, request)

Fire ChatParams hooks for request. Returns {:ok, request, state} (with any {:inject, msg} returns appended to both the request's and the state's messages) or {:halt, reason} when a hook bailed.

call(state, request, opts)

Run one provider inference through the shared instrumented path.

Types

option()

@type option() ::
  {:purpose, atom()}
  | {:starvation, :surface | :tolerate}
  | {:chat_params, boolean()}
  | {:stream_cb, (term() -> term()) | nil}

Functions

apply_chat_params(state, request)

@spec apply_chat_params(ExAthena.Loop.State.t(), ExAthena.Request.t()) ::
  {:ok, ExAthena.Request.t(), ExAthena.Loop.State.t()} | {:halt, term()}

Fire ChatParams hooks for request. Returns {:ok, request, state} (with any {:inject, msg} returns appended to both the request's and the state's messages) or {:halt, reason} when a hook bailed.

Exposed for callers that must fire hooks exactly once outside call/3 (ReAct fires them per iteration, before its transient-error retry path).

call(state, request, opts)

@spec call(ExAthena.Loop.State.t(), ExAthena.Request.t(), [option()]) ::
  {:ok, ExAthena.Response.t(), ExAthena.Loop.State.t()}
  | {:halt, term()}
  | {:error, {:error_thinking_starved, map(), ExAthena.Loop.State.t()}}
  | {:error, term()}

Run one provider inference through the shared instrumented path.

Options:

  • :purpose (required) — atom tag for telemetry attribution.
  • :starvation (required) — :surface or :tolerate; see moduledoc.
  • :chat_params — fire ChatParams hooks (default true). ReAct passes false because it applies them once per iteration itself (via apply_chat_params/2) so a transient-error retry cannot double-fire hooks or double-append {:inject, …} messages.
  • :stream_cb — provider stream callback (from ExAthena.Modes.ReAct.stream_callback/1); nil for one-shot query/2 calls.

Returns:

  • {:ok, response, state} — usage + cost folded into state.budget, {:usage, _} emitted.
  • {:halt, reason} — a ChatParams hook halted before the call.
  • {:error, {:error_thinking_starved, info, state}} — starved response under starvation: :surface; state carries the folded budget.
  • {:error, reason} — provider error, unchanged.