A circuit breaker that trips the whole cluster when any one node trips.
The default breaker (ExternalService.CircuitBreaker.Fuse) is node-local:
every node has to learn independently that a service is failing, so a cluster
of N nodes sends roughly N times the failing traffic before all of them stop.
This backend keeps that same local breaker but tells the other nodes when it
opens, so they open too.
use ExternalService,
circuit_breaker: [
tolerate: 5,
within: :timer.seconds(1),
reset: :timer.seconds(5),
backend: ExternalService.CircuitBreaker.Cluster
]Options
:nodes- which nodes to notify, as a list or a zero-arity function returning a list. Defaults to&Node.list/0(every connected node). Supply a function when only some of your cluster calls the service:backend: {ExternalService.CircuitBreaker.Cluster, nodes: &MyApp.api_nodes/0}
Every other option is passed through to the underlying :fuse breaker.
How it works
Each node keeps its own ordinary fuse. When a node's breaker transitions
from closed to open, and only then, it sends a fire-and-forget
:erpc.multicast/4 to the other nodes. A node receiving that message trips its
own breaker, which means its own reset timer then closes it on the normal
schedule — there is no distributed state to keep, no shared store to reach, and
no process or supervision tree that this library has to run.
Only locally originated trips are broadcast, so a trip costs one round of messages rather than one per node per node.
What this costs you
This backend deliberately trades isolation for speed of convergence, and that trade is not always the right one:
- One node can trip the whole cluster. If a single node has a bad network path while the service is healthy for everyone else, it will still take the cluster's breakers with it. If you would rather each node judge the service for itself — the bulkhead argument — stay on the default breaker.
- Propagation is best-effort. The multicast is fire-and-forget: a message lost to a netsplit or a slow node simply means that node keeps its own counsel until it trips on its own, which is exactly the default behavior. Nothing breaks; convergence is just slower.
- Recovery is not coordinated. Each node's reset timer runs independently. Because they trip at roughly the same moment they also reset at roughly the same moment, but they do not agree on it, and the first node to close will probe the service alone.
- A node that starts later does not catch up. It learns the service is unhealthy the first time one of its own calls fails.
Automatic resets are not broadcast (each node's timer handles its own), but an
explicit ExternalService.reset/1 is, since that is a deliberate
administrative action.