ExAgent.Models.Fallback (ExAgent v2.2.0)

View Source

Retries transient provider failures and falls back to other models.

primary = ExAgent.Models.ReqLLM.new(model: ..., foreign_history: :portable)
backup = ExAgent.Models.ReqLLM.new(model: ..., foreign_history: :portable)

model = ExAgent.Models.Fallback.new(models: [primary, backup])
ExAgent.new(model: model, tools: tools)

Each model request tries models in order. A model is retried up to :max_retries times with exponential backoff (:backoff_ms doubling up to :max_backoff_ms, with jitter) and then the next model is tried at once. Only failures that returned no output are retried or fall back:

  • ExAgent.RequestError with an HTTP status in :retry_statuses (default 408, 429, 500, 502, 503, 504 and 529) and no partial response;
  • with retry_transport: true, also connection failures (RequestError.transport set) and total timeouts. The provider may already have processed (and billed) such a request, so this is opt-in.

A gateway may still bill an upstream attempt behind a 502 or 504. Other errors, including configuration errors detected before IO, return immediately. A streaming request is retried only when its first event is a retryable error; once output has been emitted the stream is committed.

The loop counts one admitted model request however many attempts it makes. Each attempt keeps settings.timeout as its receive timeout; the run deadline (ModelRequestParameters.deadline) bounds all attempts together, so retries never outlive it. Every retry emits [:exagent, :model, :retry] telemetry.

Different models only accept each other's responses in the history when configured for it, e.g. ExAgent.Models.ReqLLM with foreign_history: :portable; otherwise switching models mid-conversation fails like any other mismatched history. A persisted history resumes when any of the models accepts it. The profile is the intersection of the models' profiles. model_name/1 and system/1 report the first model; each response keeps the name of the model that produced it. An explicit estimate_cost callback receives this fallback model, not the model that served the request, so price per response (model_name) or rely on usage reported by the models.

models must be model structs (resolve spec strings with ExAgent.Model.resolve/2 first); build the struct with new/1.

Summary

Functions

Builds a fallback model, raising ArgumentError on invalid options.

Types

t()

@type t() :: %ExAgent.Models.Fallback{
  backoff_ms: non_neg_integer(),
  max_backoff_ms: non_neg_integer(),
  max_retries: non_neg_integer(),
  models: [ExAgent.Model.model()],
  retry_statuses: [pos_integer()],
  retry_transport: boolean()
}

Functions

new(opts)

@spec new(keyword()) :: t()

Builds a fallback model, raising ArgumentError on invalid options.