Normandy.Resilience.CircuitBreaker (normandy v2.2.0)

View Source

Circuit Breaker pattern implementation to prevent cascading failures.

The circuit breaker monitors failures and can transition between three states:

  • Closed: Normal operation, requests pass through
  • Open: Failures exceeded threshold, requests fail fast
  • Half-Open: Testing if service recovered, limited requests allowed

States

Closed ──(failures >= threshold)──> Open
  ↑                                   │
  │                                   │
  └──(success)── Half-Open ←─(timeout)┘

Features

  • Automatic state transitions based on failure rates
  • Configurable failure threshold and timeout
  • Half-open state for gradual recovery
  • Thread-safe using GenServer
  • Metrics and monitoring support

Admission and stale results

A call is admitted by one atomic request to the breaker, which returns the breaker's current generation. Every state change starts a new generation, and a result is counted only in the generation it was admitted in, so a slow call admitted while closed cannot close (or reopen) a half-open breaker. A half-open probe is monitored: if its process dies before it returns, its slot is freed and it counts as a failed probe, which reopens the breaker.

Shared breakers

BaseAgent agents configured with circuit_breaker: use a breaker shared by every agent with the same client module, endpoint, model and breaker options (key/3). Normandy starts it on first use under its own supervisor (ensure_started/1), so it does not depend on the process that built the agent. Breakers are per node. Set :scope to give a tenant (or a test) its own breaker.

BaseAgent counts only failures that say the endpoint is unhealthy (see BaseAgent.init/1's :circuit_breaker); call/3 and run/3 take the same choice as :failure_if.

start_link/1 and call/3 remain for a breaker you start and supervise yourself.

Example

# Start a circuit breaker
{:ok, cb} = CircuitBreaker.start_link(
  name: :api_breaker,
  failure_threshold: 5,
  timeout: 60_000
)

# Execute protected call
case CircuitBreaker.call(cb, fn ->
  MyAPI.risky_operation()
end) do
  {:ok, result} -> handle_success(result)
  {:error, :open} -> handle_circuit_open()
  {:error, reason} -> handle_failure(reason)
end

# Check state
CircuitBreaker.state(cb)  # :closed | :open | :half_open

Summary

Types

circuit_breaker_option()
circuit_breaker_options()
key()

A shared breaker's identity: the client's module, its endpoint (scheme, host, port and path of its base_url, or nil), the model, and the breaker options with their defaults, sorted. Never a credential.

state()

Functions

call(server, fun, opts \\ [])

Execute a function protected by the circuit breaker.

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

ensure_started(key)

The running breaker for key, starting it under Normandy's supervisor when there is none. The breaker is not linked to the caller, and a breaker that crashed, or that stopped itself after being idle, is not running and is not restarted: the next ensure_started/1 starts a fresh, closed one.

fetch(key)

The running breaker for key, or :error when none runs.

key(client, model, opts)

The key of the shared breaker for client, model and the breaker options opts.

metrics(server)

Get detailed metrics about the circuit breaker.

reset(server)

Manually reset the circuit breaker to closed state.

run(key, fun, opts \\ [])

call/3 through the shared breaker for key, started if needed (ensure_started/1). Returns what call/3 returns. Takes call/3's options.

start_link(opts \\ [])

Start a circuit breaker GenServer.

state(server)

Get the current state of the circuit breaker.

trip(server)

Manually open the circuit breaker (for testing or maintenance).

Types

circuit_breaker_option()

@type circuit_breaker_option() ::
  {:name, atom()}
  | {:failure_threshold, pos_integer()}
  | {:success_threshold, pos_integer()}
  | {:timeout, pos_integer()}
  | {:half_open_max_calls, pos_integer()}

circuit_breaker_options()

@type circuit_breaker_options() :: [circuit_breaker_option()]

key()

@type key() :: {module(), String.t() | nil, String.t(), keyword()}

A shared breaker's identity: the client's module, its endpoint (scheme, host, port and path of its base_url, or nil), the model, and the breaker options with their defaults, sorted. Never a credential.

state()

@type state() :: :closed | :open | :half_open

Functions

call(server, fun, opts \\ [])

@spec call(GenServer.server(), (-> term()), keyword()) ::
  {:ok, term()} | {:error, term()}

Execute a function protected by the circuit breaker.

Returns

  • {:ok, result} - Function executed successfully
  • {:error, :open} - Circuit is open, request blocked
  • {:error, reason} - Function failed with reason; a raise is {:error, {:exception, e}}, an exit {:error, {:exit, reason}} and a throw {:error, {:throw, value}}, each recorded as a failure

Examples

CircuitBreaker.call(breaker, fn ->
  {:ok, MyAPI.call()}
end)

# With explicit error handling
CircuitBreaker.call(breaker, fn ->
  case MyAPI.call() do
    {:ok, result} -> {:ok, result}
    {:error, _} = error -> error
  end
end)

Options

  • :failure_if (function of arity 1) - Decides whether a failed call counts against the breaker. It gets the {:error, reason} the call returns (a raise, exit or throw is {:error, {:exception | :exit | :throw, _}}, a return that is neither {:ok, _} nor {:error, _} is {:error, {:unexpected_return, value}}); false leaves the counts unchanged and frees a half-open probe's slot. It must return a boolean; one that raises or returns anything else counts the failure. By default every failure counts.

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

See Supervisor.

ensure_started(key)

@spec ensure_started(key()) :: {:ok, pid()}

The running breaker for key, starting it under Normandy's supervisor when there is none. The breaker is not linked to the caller, and a breaker that crashed, or that stopped itself after being idle, is not running and is not restarted: the next ensure_started/1 starts a fresh, closed one.

fetch(key)

@spec fetch(key()) :: {:ok, pid()} | :error

The running breaker for key, or :error when none runs.

key(client, model, opts)

@spec key(struct(), String.t(), keyword()) :: key()

The key of the shared breaker for client, model and the breaker options opts.

Agents whose client module, endpoint, model and options are all equal share one breaker. The endpoint is the client's base_url (its struct field, else client.options[:base_url]) reduced to scheme, host, port and path; credentials, query and fragment are dropped.

Options

  • :failure_threshold (pos_integer/0) - Failures that open the breaker. The default value is 5.

  • :success_threshold (pos_integer/0) - Successes while half-open that close it again. The default value is 2.

  • :timeout (pos_integer/0) - Milliseconds the breaker stays open before it lets probe calls through (half-open). Not a call timeout. The default value is 60000.

  • :half_open_max_calls (pos_integer/0) - Probe calls allowed at once while half-open. The default value is 1.

  • :scope (term/0) - Any plain term that separates breakers which would otherwise share one (one per tenant, one per test). It appears in telemetry and logs: never put a secret in it. The default value is nil.

metrics(server)

@spec metrics(GenServer.server()) :: map()

Get detailed metrics about the circuit breaker.

Returns

Map with current state and statistics

Examples

CircuitBreaker.metrics(breaker)
#=> %{
  state: :closed,
  generation: 0,
  failure_count: 2,
  success_count: 100,
  failure_threshold: 5,
  success_threshold: 2,
  half_open_calls: 0,
  opened_at: nil,
  last_failure_time: nil
}

reset(server)

@spec reset(GenServer.server()) :: :ok

Manually reset the circuit breaker to closed state.

Examples

CircuitBreaker.reset(breaker)
#=> :ok

run(key, fun, opts \\ [])

@spec run(key(), (-> term()), keyword()) :: {:ok, term()} | {:error, term()}

call/3 through the shared breaker for key, started if needed (ensure_started/1). Returns what call/3 returns. Takes call/3's options.

start_link(opts \\ [])

@spec start_link(circuit_breaker_options()) :: GenServer.on_start()

Start a circuit breaker GenServer.

Options

  • :name (term/0) - A GenServer.name/0 for the breaker.

  • :failure_threshold (pos_integer/0) - Failures that open the breaker. The default value is 5.

  • :success_threshold (pos_integer/0) - Successes while half-open that close it again. The default value is 2.

  • :timeout (pos_integer/0) - Milliseconds the breaker stays open before it lets probe calls through (half-open). Not a call timeout. The default value is 60000.

  • :half_open_max_calls (pos_integer/0) - Probe calls allowed at once while half-open. The default value is 1.

Examples

{:ok, cb} = CircuitBreaker.start_link(name: :my_breaker)
{:ok, cb} = CircuitBreaker.start_link(
  name: :api_breaker,
  failure_threshold: 10,
  timeout: 30_000
)

state(server)

@spec state(GenServer.server()) :: state()

Get the current state of the circuit breaker.

Returns

  • :closed - Normal operation
  • :open - Circuit is open, failing fast
  • :half_open - Testing recovery

Examples

CircuitBreaker.state(breaker)
#=> :closed

trip(server)

@spec trip(GenServer.server()) :: :ok

Manually open the circuit breaker (for testing or maintenance).

Examples

CircuitBreaker.trip(breaker)
#=> :ok