How to protect a GenServer with a circuit breaker

This guide shows you how to wrap a GenServer with Tripwire's circuit breaker, configure its sensitivity for your dependency's behavior, and handle the different responses. You should already be familiar with OTP supervision trees, GenServer.call/3, and Mix dependency management.

1. Add Tripwire to your project

To add Tripwire as a dependency, open mix.exs and include it:

defp deps do
  [
    {:tripwire, "~> 0.1.0"}
  ]
end

Fetch the dependency:

mix deps.get

2. Set up the infrastructure

Tripwire requires two processes to be running in your supervision tree before any breakers can start.

To set up the infrastructure, add these to your Application.start/2 children list:

children = [
  # A Registry maps target keys to breaker processes.
  {Registry, keys: :unique, name: Tripwire.Breaker.registry()},
  # A TaskSupervisor handles non-blocking call forwarding.
  {Task.Supervisor, name: Tripwire.TaskSupervisor}
]

The Registry must be named :tripwire_breakers and the TaskSupervisor must be named Tripwire.TaskSupervisor -- these names are hardcoded in the Breaker module. If you already have a Registry or Task.Supervisor with these exact names in your tree, you can skip adding them again.

3. Add a breaker for each target

To wrap a GenServer with a circuit breaker, add a Tripwire child to your supervision tree alongside the target:

children = [
  {Registry, keys: :unique, name: Tripwire.Breaker.registry()},
  {Task.Supervisor, name: Tripwire.TaskSupervisor},
  # Your target GenServer, registered by name so the breaker can reach it.
  {MyApp.Dependency, [name: MyApp.Dependency]},
  # The circuit breaker for this target.
  {Tripwire, target_key: :my_dependency, target_pid: MyApp.Dependency}
]

The target_key is your chosen identifier for this dependency. You use it when calling through the breaker. The target_pid is the PID or registered name of the GenServer to protect.

If you are retrofitting an existing GenServer that already runs in your supervision tree without a registered name, update its child spec to include a name:

# Before (no name):
{MyApp.Dependency, []}
# After (registered name):
{MyApp.Dependency, [name: MyApp.Dependency]}

You can start multiple breakers in the same tree, one per dependency:

children = [
  {Registry, keys: :unique, name: Tripwire.Breaker.registry()},
  {Task.Supervisor, name: Tripwire.TaskSupervisor},
  {MyApp.Database, [name: MyApp.Database]},
  {MyApp.Cache, [name: MyApp.Cache]},
  {Tripwire, target_key: :database, target_pid: MyApp.Database},
  {Tripwire, target_key: :cache, target_pid: MyApp.Cache}
]

Each breaker operates independently. A failure in the database dependency does not affect the cache dependency.

4. Call through the breaker

To make a call through the breaker, use Tripwire.call/3 with the target key you chose:

Tripwire.call(:my_dependency, {:query, "SELECT 1"})

If you are retrofitting existing code, replace direct GenServer.call calls with Tripwire.call, mapping the PID or registered name to a target key:

# Before:
GenServer.call(MyApp.Dependency, {:query, "SELECT 1"})

# After:
Tripwire.call(:my_dependency, {:query, "SELECT 1"})

You can pass a custom timeout as the third argument:

Tripwire.call(:my_dependency, {:query, "SELECT 1"}, 10_000)

The timeout applies to the call to the target, not to the breaker. If the target does not respond within the timeout, Tripwire counts it as a failure.

5. Adjust the failure threshold

The failure threshold controls how many consecutive failures must occur before the circuit opens. The default is 5.

To adjust the threshold, pass the :threshold option when starting the breaker:

{Tripwire,
 target_key: :my_dependency,
 target_pid: MyApp.Dependency,
 threshold: 3}

If your dependency has transient failures that resolve quickly (e.g., network blips, temporary queue congestion), raise the threshold to avoid tripping the circuit unnecessarily:

{Tripwire,
 target_key: :noisy_dependency,
 target_pid: MyApp.Noisy,
 threshold: 10}

If your dependency fails hard and fast when it does fail (e.g., a process that crashes on any error), lower the threshold to protect callers sooner:

{Tripwire,
 target_key: :brittle_dependency,
 target_pid: MyApp.Brittle,
 threshold: 2}

See the Tripwire module documentation for the full list of options.

6. Adjust the recovery timeout

The recovery timeout controls how long the circuit stays open before transitioning to half-open and allowing a probe call. The default is 60,000 milliseconds (1 minute).

To adjust the recovery timeout, pass the :reset_timeout option:

{Tripwire,
 target_key: :my_dependency,
 target_pid: MyApp.Dependency,
 threshold: 5,
 reset_timeout: 30_000}

If your dependency is a local GenServer that restarts quickly under its supervisor, a short timeout (5-10 seconds) lets it recover promptly:

{Tripwire,
 target_key: :local_service,
 target_pid: MyApp.Local,
 reset_timeout: 5_000}

If your dependency connects to an external service that takes time to recover (e.g., a database connection pool that needs to drain and rebuild), use a longer timeout:

{Tripwire,
 target_key: :external_api,
 target_pid: MyApp.ApiClient,
 reset_timeout: 120_000}

7. Handle circuit breaker responses

Tripwire.call/3 returns different tuples depending on the circuit state and the target's behavior. To handle each case, pattern match on the return value:

case Tripwire.call(:my_dependency, :ping) do
  {:ok, result} ->
    # The target responded successfully. Forward the result.
    result

  {:error, :circuit_open} ->
    # The circuit is open. Do not call the target.
    # Use a fallback, return a cached value, or retry later.
    {:error, :dependency_unavailable}

  {:error, :breaker_not_registered} ->
    # No breaker was started for this target key.
    # Likely a configuration error or the breaker process crashed.
    {:error, :breaker_missing}

  {:error, :breaker_busy} ->
    # The breaker process did not respond within the caller's
    # timeout. This indicates the breaker is overloaded or stuck.
    {:error, :breaker_unreachable}

  {:error, {:breaker_crashed, reason}} ->
    # The breaker process crashed between starting and responding.
    # The supervisor will restart it. Retry the call.
    {:error, {:breaker_down, reason}}
end

If you only care about the success case and want to treat all errors uniformly, use a catch-all:

case Tripwire.call(:my_dependency, :ping) do
  {:ok, result} -> {:ok, result}
  _ -> {:error, :service_unavailable}
end

What you have

You have protected a GenServer with Tripwire's circuit breaker. The breaker detects consecutive failures, blocks calls while the target is unhealthy, and automatically probes for recovery. You have tuned the threshold and timeout for your dependency's specific behavior, and your callers handle both the success and failure paths.

To learn how Tripwire's state machine works under the hood, see the Tripwire.State module documentation.