# Retries Many failures when calling an external service are _transient_: a momentary timeout, a brief overload, a connection reset. The simplest effective response is to try again — perhaps after a short backoff. `ExternalService` automates this using the [retry](https://hex.pm/packages/retry) library, exposing its flexibility through the `ExternalService.RetryOptions` struct. ## Triggering a retry Inside the function you pass to `call`, you signal that a retry should happen by returning either: - the atom `:retry`, or - a tuple `{:retry, reason}`, where `reason` is any term. Anything else is a success and is returned to the caller as-is — including the function's own `{:error, reason}` results. You decide what is retriable: ```elixir call fn -> case HTTP.post(url, body) do {:ok, %{status: 200} = resp} -> {:ok, resp} {:ok, %{status: s}} when s in 500..599 -> {:retry, s} # retry server errors {:ok, %{status: 429}} -> :retry # retry throttling {:ok, %{status: 4xx}} = resp -> resp # client error: don't retry {:error, reason} -> {:error, reason} end end ``` Each retry melts the service's circuit breaker, so a sustained run of retries will eventually open it. See [Circuit breakers](circuit-breakers.md). ## Retrying on the return value with `:retry_on` Returning `:retry` works when you control the function body. When you're calling an existing function that already returns its own result shape — and you'd rather not wrap it just to translate that shape into `:retry` — the `:retry_on` retry option takes a **predicate** that is run on the return value. When the predicate returns a truthy value, the call is retried exactly as if the function had returned `{:retry, result}`: the result becomes the retry reason, and the circuit breaker melts. ```elixir # Retry any 5xx response from an unmodified client function. retry: [retry_on: &match?({:error, %{status: s}} when s in 500..599, &1)] call fn -> Stripe.charge(params) end ``` If retries are exhausted, the matched result is carried as the reason on `ExternalService.RetriesExhausted` (and in the `[:external_service, :call, :retry]` telemetry metadata). An explicit `:retry` / `{:retry, reason}` return from the function always takes precedence over the predicate. > #### Prefer explicit returns when you own the function {: .tip} > > `:retry_on` is for adapting functions you don't want to change. When you do > control the body, returning `:retry` / `{:retry, reason}` keeps the retry > decision explicit and local to the call. ## Configuring retries Retry behavior is described by `ExternalService.RetryOptions`. You can supply it as the service's default (the `:retry` option to `use ExternalService` / `start/2`), or per call as a keyword list or struct: ```elixir # Service default use ExternalService, retry: [max_attempts: 5, backoff: :exponential, base: 100, jitter: true] # Per-call override (keyword list) call [max_attempts: 2, backoff: :linear, base: 50], fn -> work() end # Per-call override (struct) call %ExternalService.RetryOptions{max_attempts: 2}, fn -> work() end ``` When you use the two-argument `call/2` (no options), the service's default `:retry` options apply. > #### Per-call keyword lists merge; structs replace {: .info} > > A per-call **keyword list** is treated as a set of *overrides*: it is merged > onto the service's configured `:retry` defaults, changing only the keys you > list and inheriting the rest. So if a service is configured with > `retry: [backoff: :exponential, base: 100, max_attempts: 5]`, then > `call([max_attempts: 2], fun)` runs with `backoff: :exponential, base: 100, > max_attempts: 2`. > > A per-call **`%RetryOptions{}` struct**, by contrast, is already a complete set > of options, so it *replaces* the service defaults wholesale — any field you > don't set takes the library default, not the service's value. ### The options | Option | Default | Meaning | | --------------- | -------------- | -------------------------------------------------------------------------------------------- | | `:backoff` | `:exponential` | Growth strategy for the delay between retries: `:exponential` or `:linear`. | | `:base` | `10` | Initial delay between retries, in milliseconds (`0` for no delay). | | `:factor` | `1` | Growth factor applied each retry. Only used for `:linear` backoff. | | `:cap` | — | Caps the delay between retries to at most this many milliseconds. | | `:expiry` | — | Total time budget for retries, in milliseconds. Retrying stops once exceeded. | | `:max_attempts` | — | Maximum number of attempts (initial plus retries). No limit by default. | | `:jitter` | `false` | Random jitter on delays. `true` applies ±10%; a float (e.g. `0.25`) applies that proportion. | | `:retry_on` | — | Predicate run on the return value; retry when it returns a truthy value (see below). | | `:retry_exceptions` | `[]` | Which raised exceptions trigger a retry: a list of exception modules, or a predicate run on the exception (see below). | Options are validated when the struct is built; an invalid value raises `NimbleOptions.ValidationError` with a helpful message. ## Backoff strategies **Exponential** backoff grows the delay multiplicatively, starting from `:base`. This is the right default for most services: it backs off quickly when a service is struggling. ```elixir retry: [backoff: :exponential, base: 100] # delays grow ~100ms, 200ms, 400ms, 800ms, ... ``` **Linear** backoff grows the delay by `:factor` each time, starting from `:base`: ```elixir retry: [backoff: :linear, base: 100, factor: 1] # delays grow ~100ms, 200ms, 300ms, 400ms, ... ``` ## Bounding retries By default there is **no** `:max_attempts`, `:expiry`, or `:cap`, so returning `:retry` repeatedly keeps retrying with an ever-growing delay. You almost always want an explicit bound. There are two, and they compose: - **`:max_attempts`** — a count. `max_attempts: 5` means at most five attempts total (the first try plus four retries). - **`:expiry`** — a time budget in milliseconds. Once cumulative retry time exceeds it, retrying stops. It is checked **between** attempts, so it bounds when the next attempt starts, never how long the current one runs — see [Nothing here bounds a single attempt](#nothing-here-bounds-a-single-attempt). You can use either or both; whichever is reached first stops the retries. When the bound is hit without success, `call/3` returns `{:error, %ExternalService.RetriesExhausted{}}` (and `call!/3` raises it). ```elixir # Stop after 5 attempts OR 5 seconds, whichever comes first. retry: [max_attempts: 5, expiry: :timer.seconds(5), backoff: :exponential, base: 100] ``` ### Nothing here bounds a single attempt `:expiry` reads like a wall-clock budget for the whole call. It isn't, and the difference matters as soon as your function is slow rather than fast-failing. Both bounds are evaluated **between** attempts — when the retry machinery decides whether to make another one — so neither can interrupt an attempt already running. Measured with `retry: [max_attempts: 4, expiry: 100]` against a function that sleeps 300ms and then returns `:retry`: ``` attempts made: 2 total elapsed: 706ms ``` The 100ms budget stopped the third attempt, but the call still took seven times the budget, because each attempt ran to completion first. Push that further — a function that never returns — and `:expiry` is never evaluated at all, so the call hangs forever no matter what you set. **`ExternalService` imposes no timeout on your function.** Bounding a single attempt is the caller's job; see [When the service hangs](circuit-breakers.md#when-the-service-hangs). > #### Don't rely on the circuit breaker to bound retries {: .warning} > > The breaker is a backstop, not a retry bound. With no `:max_attempts`/`:expiry`, > retries stop only when the breaker opens — and that is not guaranteed. The > breaker opens after `:tolerate` failures *within* its `:within` window, but > exponential backoff keeps widening the gap between attempts. Once the delay > grows past the window, failures stop accumulating fast enough to trip the > breaker, and retrying continues indefinitely. > > This needs no exotic configuration. With a **fully default** circuit breaker > (`tolerate: 10`, `within: 10_000`) and retry options default except for the > `base: 100` used throughout this guide, attempts land at roughly 0ms, 100ms, > 300ms, 700ms, 1.5s, 3.1s, 6.3s, 12.7s, … Only seven of them ever fall inside > any 10-second window, so the melt count tops out at 7 against a `:tolerate` of > 10. The breaker never opens and the call never returns. Always set an explicit > `:max_attempts` or `:expiry` — and a `:cap`, below — for unattended retries. > > The coupling runs the other way too: every failing attempt melts the breaker, > so a bounded `:max_attempts` makes the breaker open *sooner* than `:tolerate` > suggests. See > [`:tolerate` counts attempts, not calls](circuit-breakers.md#what-counts-as-a-failure). ### Unbounded retries are a choice, not a default to fall into Because leaving both bounds unset is so rarely what you want, `ExternalService.start/2` logs a warning when a service's default retry options set neither: ``` [warning] ExternalService.start(:my_service, ...) sets no retry bound: neither :max_attempts nor :expiry is configured, so a call that keeps returning :retry will retry forever. ... ``` If unbounded retrying really is what you want — a background job that should keep trying until it succeeds, say — or if every call site supplies its own bound, set a bound to `:infinity` to say so. It behaves exactly like leaving the bound unset and silences the warning: ```elixir ExternalService.start(:my_service, retry: [max_attempts: :infinity]) ``` Even then, pair it with a `:cap` so the delay between attempts doesn't grow without limit. ## Capping the delay Exponential backoff grows without bound. `:cap` puts a ceiling on any single delay so you don't end up waiting minutes between attempts: ```elixir retry: [backoff: :exponential, base: 100, cap: :timer.seconds(2)] # delays grow 100, 200, 400, 800, 1600, 2000, 2000, ... (capped at 2s) ``` ## Jitter When many processes retry on the same schedule, they retry in lockstep and slam the recovering service all at once — the _thundering herd_. Jitter randomizes each delay to spread them out: ```elixir retry: [backoff: :exponential, base: 100, jitter: true] # ±10% retry: [backoff: :exponential, base: 100, jitter: 0.25] # ±25% ``` Enabling jitter is good practice for any service with many concurrent callers. ## Retrying on raised exceptions **By default, raised exceptions are not retried** — they propagate straight to the caller. This is a deliberate 2.0 change (see [issue #7](https://github.com/jvoegele/external_service/issues/7)): retrying every `RuntimeError` by default tended to mask real bugs. If a particular exception genuinely indicates a transient condition worth retrying, list its module in `:retry_exceptions`: ```elixir retry: [retry_exceptions: [MyApp.TransientError, DBConnection.ConnectionError]] ``` Now a raised `MyApp.TransientError` triggers a retry just like a `:retry` return value would, and it melts the circuit breaker. Exceptions not in the list still propagate untouched and leave the breaker alone — `:retry_exceptions` governs both retrying and whether a raised exception counts against the breaker. If retries run out while retrying an exception, that original exception is re-raised — with its original stacktrace, still pointing at the code that raised it — rather than being converted into an `ExternalService.RetriesExhausted`. ### Deciding per exception rather than per type A module list settles the question by type, which is the wrong grain when the same exception is transient in one instance and permanent in another — an HTTP client that raises one error struct for every status, say. Give `:retry_exceptions` a predicate instead and it is run on the exception itself: ```elixir retry: [ retry_exceptions: fn %MyApp.HTTPError{status: status} -> status >= 500 _other -> false end ] ``` A truthy return retries and melts the breaker; anything else propagates the exception untouched, exactly as an unlisted module would. The predicate replaces the list rather than supplementing it, so fold any module checks you still want into it: > #### Predicates must answer for every value {: .info} > > A predicate is handed every exception the call raises, including ones it does > not recognise, so a `%MyApp.HTTPError{}`-only clause will eventually meet > something else. A predicate that fails rather than answering — raising, > throwing, or exiting — is treated as **no match**: the exception it was asked > to classify reaches the caller unchanged, nothing is retried, the breaker is > left alone, and a warning naming the option and the service is logged. The same > holds for the `:retry_on` predicate, where the call's result passes through > untouched. A broken classifier never changes the outcome of a call; it only > stops retries from happening. ```elixir retry_exceptions: fn %MyApp.HTTPError{status: status} -> status >= 500 error -> is_struct(error, DBConnection.ConnectionError) end ``` > #### Anonymous functions and `use ExternalService` {: .warning} > > The examples above pass an anonymous function, which works for > `ExternalService.start/2` and for per-call retry options. The options given to > `use ExternalService` are stored in a module attribute, though, which cannot > hold one: > > ```elixir > use ExternalService, retry: [retry_exceptions: fn error -> ... end] > # ** (ArgumentError) cannot inject attribute @__external_service_opts__ into > # function/macro because cannot escape #Function<...> > ``` > > Put the predicate in a named function and pass a remote capture — > `&MyApp.Retry.transient?/1` — which is a valid attribute value. The same > applies to the `:retry_on` predicate. This pairs well with [Errata](https://hexdocs.pm/errata), whose errors classify themselves — an error type can decide from its own `:reason` or `:context` and answer through `Errata.retryable?/1`: ```elixir defmodule MyApp.Retry do require Errata def retryable_error?(error), do: Errata.is_error(error) and Errata.retryable?(error) end # ...then, in the service configuration: retry: [retry_exceptions: &MyApp.Retry.retryable_error?/1] ``` See [Using Errata in Your Application](errata.md) for both boundaries together, and for the difference between an error being retryable and a call being safe to repeat. > #### Prefer return values over exceptions {: .tip} > > Where you can, drive retries with `:retry` / `{:retry, reason}` return values > (or the `:retry_on` predicate) rather than relying on `:retry_exceptions`. It > keeps the retry decision explicit and local to the call, and avoids retrying an > exception that happens to share a type with a genuine bug. ## Putting it together A solid default for an HTTP-style dependency: ```elixir use ExternalService, circuit_breaker: [tolerate: 5, within: :timer.seconds(1), reset: :timer.seconds(5)], retry: [ backoff: :exponential, base: 100, cap: :timer.seconds(2), max_attempts: 5, expiry: :timer.seconds(10), jitter: true ] ``` This retries transient failures with jittered exponential backoff, never waits more than 2 seconds between attempts, gives up after 5 attempts or 10 seconds, and lets the circuit breaker take over if the failures are sustained.