About Tripwire's Architecture

This document explains why Tripwire is built the way it is. It addresses the design decisions behind the three-state circuit breaker, the module structure, the use of async forwarding, and the choice of BEAM primitives. It assumes familiarity with GenServer, supervision trees, and the circuit breaker pattern.

This explanation does not cover how to use Tripwire (see the getting started tutorial and the how-to guide), nor does it enumerate the API surface (see the Tripwire module documentation). It does not compare Tripwire to alternative libraries or discuss the general circuit breaker pattern in distributed systems.

Why cascading failure needs a circuit breaker

When one GenServer depends on another, each GenServer.call/3 creates a synchronous coupling. The caller blocks until the callee responds. If the callee is healthy, this is efficient. If the callee degrades, the caller blocks waiting for a timeout. If the caller is itself a GenServer serving other callers, the degradation propagates.

This is cascading failure. A single degraded process blocks its callers, which block their callers, and the entire call tree collapses into timeout soup. The BEAM's process isolation prevents one degraded process from crashing the system, but it does not prevent the caller from waiting.

Tripwire breaks this chain by wrapping the call. Instead of calling the target directly, the caller calls through Tripwire. When Tripwire detects that the target is failing, it stops forwarding calls and returns immediately. The caller stays responsive. The call chain stays intact.

Why three states

Tripwire uses three circuit states: :closed, :open, and :half_open. Each state exists because of a different constraint on the problem.

closed exists because most of the time, dependencies are healthy. The circuit breaker's default behavior should be to forward calls. If the circuit started in any other state, the caller would need to explicitly transition to normal operation, violating the goal of zero caller boilerplate.

open exists because once a dependency is known to be failing, further calls are wasteful. Each call to a failing dependency blocks for its full timeout duration. If the caller makes calls faster than they time out, the caller's mailbox fills with blocked callers. Opening the circuit stops this. The fast-fail response ({:error, :circuit_open}) takes microseconds, not seconds.

half_open exists because recovery is unknowable without probing. A remote process's internal state is not observable from outside. The only way to discover whether a dependency has recovered is to try a call. But trying a call on every request while the dependency is likely still degraded would repeat the cascading failure. The half-open state resolves this tension. After the recovery timer fires, exactly one call is allowed through. If it succeeds, the circuit closes. If it fails, the circuit reopens and the timer restarts. This single-probe design means the breaker tests recovery without flooding the dependency with doomed calls.

These three states trace to three immovable constraints. Transient failures are inherent in any communication channel. A single failure does not mean the dependency is permanently broken, so the threshold exists. Recovery has non-zero duration. The reset timeout gives the dependency time to recover before probing. Recovery is epistemically opaque. The only way to know is to try, so the half-open probe exists.

Two states would be insufficient. A two-state breaker (healthy/failed) either never probes for recovery (permanent failure) or probes on every request (defeats the point of opening). Four states would be redundant. Adding a second probe state or a cooldown state would introduce complexity without addressing any constraint that the three-state model does not already handle.

Why three modules

Tripwire is split across three modules: Tripwire (public API), Tripwire.Breaker (per-target GenServer), and Tripwire.State (pure state machine). This separation is not cosmetic. It follows the Design DAG: every dependency arrow points from a more volatile module to a less volatile one.

Tripwire.State is the stable root. It contains the transition rules for the circuit breaker. It defines what happens when a failure is recorded, when the recovery timer fires, when a probe succeeds or fails. These rules reflect the circuit breaker pattern itself, which is well-established and does not change. The module is pure. It has no side effects, no process state, no I/O. Every function returns deterministically from its inputs. This makes it trivial to test in isolation.

Tripwire.Breaker depends on State. It is a GenServer that holds the current circuit state, the failure count, the recovery timer reference, and the target PID. When an event occurs, such as a call result or a timer firing, it delegates to Tripwire.State.transition/3 to compute the next state and the required side effects. The Breaker executes those side effects (scheduling or cancelling timers) but does not decide what they should be. This is the control plane.

Tripwire depends on Breaker. It is the single door through which callers enter. It handles address resolution (looking up the breaker by target key via Registry), forwards calls, and maps exit signals into caller-friendly error tuples. It does not hold state. It does not make decisions. It is the thinnest possible shell.

This structure has a specific benefit. When a Breaker process crashes due to corrupted state, the supervisor restarts it at :closed with zero failure count. The State module, the transition rules, is unaffected by the crash. The API module, the caller's entry point, continues to work for other targets. The failure is isolated to one breaker for one dependency.

Why async forwarding

When the circuit is closed, the Breaker must forward the caller's message to the target GenServer and return the target's response. A straightforward approach would be for the Breaker to call GenServer.call(target_pid, msg, timeout) directly inside its handle_call/3 callback.

This is the wrong approach. handle_call/3 runs inside the Breaker process. If the Breaker makes a synchronous call to the target and the target does not respond, the Breaker blocks. While blocked, the Breaker cannot handle recovery timer messages, cannot respond to other callers, and cannot process any other call. The Breaker has become part of the problem it was designed to solve.

Tripwire avoids this by separating the control plane from the data plane. The Breaker process (control plane) never calls the target directly. Instead, it delegates each call to a task process managed by Task.Supervisor.async_nolink/2. The task makes the call to the target and sends the result back to the Breaker as a message. The Breaker processes the message in handle_info/2, updates its state, and replies to the original caller via GenServer.reply/2.

This design has two consequences. First, the Breaker does not block on the target. Even if the target hangs, the Breaker remains responsive. It can receive timer messages, process :DOWN signals from crashing tasks, and reply to new callers with {:error, :circuit_open} if the circuit is open. Second, the caller's timeout controls how long the task waits for the target. Because the Breaker delegates the call and remains responsive to other messages regardless, the timeout is a constraint on the target, not on the Breaker.

The tradeoff is complexity. Each forwarded call creates a task, a monitor, and an asynchronous message exchange. The Breaker's state includes task_refs, a map tracking which task belongs to which caller, and probe_in_flight for the half-open probe. This is more complex than a direct synchronous call. The complexity is justified because the alternative (blocking the Breaker) defeats the purpose of the circuit breaker.

Why these BEAM primitives

Tripwire uses four BEAM primitives: GenServer, Registry, Task.Supervisor, and Process.send_after/3. Each was chosen over alternatives after considering the tradeoffs.

Registry over direct PID tracking or ETS. The Breaker must be addressable. If the caller tracked the Breaker's PID directly, the PID would become stale if the Breaker crashed and restarted. PID tracking shifts the burden of resilience onto the caller. ETS can store PID mappings durably, but ETS tables survive process restarts. A restarted Breaker would inherit an old failure count and circuit state, which violates the crash-recovery contract (restart at :closed with zero failures). Registry, used with {:via, Registry, ...} tuples, re-resolves the PID on every call. When a Breaker restarts, it re-registers under the same key, and callers transparently route to the new PID. No stale PIDs, no inherited state, no caller burden.

Task.Supervisor over raw spawn or Task.async. Calls must be forwarded without blocking the Breaker. Raw spawn/1 or Task.async/1 would create unmanaged processes with no lifecycle. If a task process hangs, there is no mechanism to terminate it. If the system runs out of process limits, there is no backpressure. Task.Supervisor provides process lifecycle management. Tasks run under a supervisor that can enforce limits, provides a namespace for debugging, and ensures cleanup on termination. The choice of async_nolink rather than async is deliberate. async would link the task to the Breaker, so a crashing task would crash the Breaker. async_nolink gives the Breaker a monitor instead, which it uses to detect and count the failure without crashing.

Process.send_after/3 over :timer.send_after/2. Both can schedule a message after a delay. Process.send_after/3 returns a reference that can be cancelled, which is necessary for the Breaker's terminate/2 callback. :timer.send_after/2 is older and less commonly used in modern Elixir. The choice between them is minor, but Process.send_after/3 is the more idiomatic option in current Elixir codebases.

GenServer over Agent or :gen_statem. The Breaker holds state (failure count, circuit state, timer refs, task refs) and responds to synchronous calls and asynchronous messages. Agent provides a simpler state-holder API but exposes state directly. The Breaker would lose control over the state transition logic. :gen_statem is Erlang's state machine behaviour and could model the three-state circuit naturally, but it introduces an Erlang-centric API that most Elixir developers do not encounter. GenServer is the idiomatic choice for Elixir. It handles synchronous calls (handle_call/3), asynchronous messages (handle_info/2), and process lifecycle (init/1, terminate/2) with patterns every Elixir developer recognizes.

What was deliberately excluded

Every library is defined as much by what it excludes as by what it includes. Several features were considered and rejected during Tripwire's design.

Retry logic. Tripwire does not automatically retry failed calls. Retry with backoff is a separate concern from circuit breaking. A circuit breaker decides whether a call should be made at all. A retry mechanism decides whether a failed call should be repeated. Conflating the two creates confused attribution. Was the call successful, or was the successful call a retry of a failure that Tripwire would have blocked? There are separate retry libraries for Elixir. Tripwire's responsibility ends at the decision to forward or block.

Fallback values. Tripwire does not substitute cached or default values when the circuit is open. Silent substitution creates a hazard. The caller cannot distinguish a real result from a fallback. If the caller depends on the accuracy of the result (a database query, a payment gate response), a substituted value is worse than a failure. It produces incorrect behavior that may go undetected. The {:error, :circuit_open} return value forces the caller to decide how to handle unavailability. That decision belongs to the caller, not to the circuit breaker.

HTTP-level health checks. Tripwire operates at the GenServer level. Adding HTTP health checks would pull in HTTP client dependencies (Mint, Finch, Hackney) that have no relationship to the core problem of GenServer call protection. It would also blur the boundary between process-level health and application-level health. A GenServer can be alive and responsive while the external service it proxies is degraded. That distinction is outside Tripwire's remit.

Dependency discovery. Tripwire does not scan the supervision tree to find GenServers that need protection. Auto-discovery requires process introspection, which violates BEAM process isolation. It also creates a hidden API surface. The caller cannot see which dependencies are protected because the list is computed at runtime. Explicit registration via start_link/1 makes the protection boundary visible and auditable.

Distributed circuit state. Tripwire does not coordinate circuit state across nodes. A circuit breaker on node A knows only about calls originating from node A. Cross-node coordination would require consensus, introduce partition tolerance issues, and allow stale state from one node to influence decisions on another. The complexity would dwarf the value. A distributed circuit breaker is a different problem from a local one, and Tripwire solves the local case.

Telemetry. Tripwire does not emit Telemetry events. Telemetry is a cross-cutting concern that belongs to the application, not the library. The same event structure that suits one application's dashboard may be wrong for another's alerting pipeline. Telemetry hooks can be added in a future release once real-world usage patterns establish what events are worth emitting.

Connections

The getting started tutorial walks through a complete Tripwire integration. After completing it, the architectural decisions described here, including why the tutorial registers a Weather GenServer by name, why it adds both a Registry and a TaskSupervisor, and why the recovery timer fires asynchronously, should make sense in context.

The how-to guide provides practical recipes for configuring thresholds, adjusting timeouts, and handling return values. The explanations here, including why the threshold exists, why the timeout matters, and why {:error, :circuit_open} is returned, provide the background for those decisions.

The Tripwire.State module documentation describes the state machine's valid transitions. The explanation here, including why each transition exists and what constraint it addresses, provides the conceptual model behind those rules.