ExternalService.CircuitBreaker behaviour (ExternalService v2.4.0)

Copy Markdown View Source

The behaviour implemented by circuit breaker backends.

A service's breaker is chosen with the :backend circuit breaker option, and defaults to ExternalService.CircuitBreaker.Fuse — a node-local breaker built on the :fuse library.

Writing a backend

defmodule MyApp.CircuitBreaker do
  @behaviour ExternalService.CircuitBreaker

  @impl true
  def install(service, options) do
    {:ok, %{key: service, tolerate: options[:tolerate]}}
  end

  @impl true
  def ask(_service, config), do: if MyStore.open?(config.key), do: :blown, else: :ok

  @impl true
  def melt(_service, config), do: MyStore.record_failure(config.key)

  @impl true
  def reset(_service, config), do: MyStore.close(config.key)

  @impl true
  def remove(_service, config), do: MyStore.forget(config.key)
end

Backends are stateless modules. install/2 returns an opaque config term that is stored with the rest of the service state (in :persistent_term) and handed back to every other callback, so a backend needs no process, supervisor, or registry of its own.

Driving a breaker directly

Most of the time the breaker is driven for you: ExternalService.call/3 asks it before each attempt and melts it on failure. The functions in this module are for the cases that fall outside a guarded call.

The one that matters is melt/1 — recording a failure the library never saw:

# A streaming connection to the service dropped, or a webhook timed out.
# That is a real failure, but it did not happen inside `call/3`.
ExternalService.CircuitBreaker.melt(:payments)

Melting counts toward the service's configured :tolerate exactly as an in-call failure does, so enough out-of-band failures will open the breaker — and, with ExternalService.CircuitBreaker.Cluster, open it across the cluster.

ask/1 and reset/1 are also here, though ExternalService.available?/1, ExternalService.blown?/1, and ExternalService.reset/1 say the same thing more readably and are usually the better call.

Summary

Types

Backend-private state, produced by install/2 and passed to every other callback.

t()

An installed circuit breaker: the backend module paired with its config.

Callbacks

Reports whether the breaker will currently admit a call.

Installs the circuit breaker for service.

Records a single failure against the breaker.

Tears the breaker down. Must be safe to call more than once.

Closes the breaker and discards its recorded failures.

Functions

Reports whether service's breaker will currently admit a call.

Records a single failure against service's breaker.

Closes service's breaker and discards its recorded failures.

Types

config()

@type config() :: term()

Backend-private state, produced by install/2 and passed to every other callback.

service()

@type service() :: ExternalService.service()

t()

@type t() :: {module(), config()}

An installed circuit breaker: the backend module paired with its config.

Callbacks

ask(service, config)

@callback ask(service(), config()) :: :ok | :blown

Reports whether the breaker will currently admit a call.

A breaker that does not exist (for example because the service was stopped while a call was in flight) must be reported as :blown rather than raising.

install(service, options)

@callback install(service(), options :: keyword()) :: {:ok, config()}

Installs the circuit breaker for service.

Receives the validated :circuit_breaker options with any backend-specific options merged in, and returns the backend's config term.

melt(service, config)

@callback melt(service(), config()) :: :ok

Records a single failure against the breaker.

remove(service, config)

@callback remove(service(), config()) :: :ok

Tears the breaker down. Must be safe to call more than once.

reset(service, config)

@callback reset(service(), config()) :: :ok | {:error, :not_found}

Closes the breaker and discards its recorded failures.

Functions

ask(service)

@spec ask(service()) :: :ok | :blown | :not_started

Reports whether service's breaker will currently admit a call.

Answers :not_started for a service that has not been started with ExternalService.start/2, which is why this is three-valued where ExternalService.available?/1 is a boolean.

melt(service)

@spec melt(service()) :: :ok | {:error, :not_found}

Records a single failure against service's breaker.

Use this to report a failure that happened outside a guarded call, so that it counts toward the breaker in the same way an in-call failure would. Enough melts within the configured :within window will open the breaker.

reset(service)

@spec reset(service()) :: :ok | {:error, :not_found}

Closes service's breaker and discards its recorded failures.

ExternalService.reset/1 is the same operation under a friendlier name.