ExAgent. Models. Fallback
(ExAgent v2.2.0)
View Source
Retries transient provider failures and falls back to other models.
primary = ExAgent.Models.ReqLLM.new(model: ..., foreign_history: :portable)
backup = ExAgent.Models.ReqLLM.new(model: ..., foreign_history: :portable)
model = ExAgent.Models.Fallback.new(models: [primary, backup])
ExAgent.new(model: model, tools: tools)Each model request tries models in order. A model is retried up to
:max_retries times with exponential backoff (:backoff_ms doubling up to
:max_backoff_ms, with jitter) and then the next model is tried at once. Only
failures that returned no output are retried or fall back:
ExAgent.RequestErrorwith an HTTP status in:retry_statuses(default 408, 429, 500, 502, 503, 504 and 529) and no partial response;- with
retry_transport: true, also connection failures (RequestError.transportset) and total timeouts. The provider may already have processed (and billed) such a request, so this is opt-in.
A gateway may still bill an upstream attempt behind a 502 or 504. Other errors, including configuration errors detected before IO, return immediately. A streaming request is retried only when its first event is a retryable error; once output has been emitted the stream is committed.
The loop counts one admitted model request however many attempts it makes.
Each attempt keeps settings.timeout as its receive timeout; the run
deadline (ModelRequestParameters.deadline) bounds all attempts together, so
retries never outlive it. Every retry emits [:exagent, :model, :retry]
telemetry.
Different models only accept each other's responses in the history when
configured for it, e.g. ExAgent.Models.ReqLLM with foreign_history: :portable; otherwise switching models mid-conversation fails like any other
mismatched history. A persisted history resumes when any of the models
accepts it. The profile is the intersection of the models' profiles.
model_name/1 and system/1 report the first model; each response keeps
the name of the model that produced it. An explicit estimate_cost
callback receives this fallback model, not the model that served the
request, so price per response (model_name) or rely on usage reported by
the models.
models must be model structs (resolve spec strings with
ExAgent.Model.resolve/2 first); build the struct with new/1.
Summary
Functions
Builds a fallback model, raising ArgumentError on invalid options.
Types
@type t() :: %ExAgent.Models.Fallback{ backoff_ms: non_neg_integer(), max_backoff_ms: non_neg_integer(), max_retries: non_neg_integer(), models: [ExAgent.Model.model()], retry_statuses: [pos_integer()], retry_transport: boolean() }
Functions
Builds a fallback model, raising ArgumentError on invalid options.