ADR-0009 — Fail-closed supervision and streaming liveness

Copy Markdown View Source

Status: Accepted (posture since C1; recorded 2026-08-23) · Related: ADR-0003 (what halts), ADR-0008 (the socket owner)

Context

Every condition that could silently lose or corrupt data halts the pipeline (ADR-0001/0003). Halting is only honest if the halt is LOUD, TERMINAL, and cannot become a livelock or take down the host application. Separately, a streaming connection's failure mode is the silent half-open partition: the reader blocks on recv forever while everything looks healthy — and MySQL only heartbeats an IDLE stream, so an active writer elsewhere does not prove OUR stream is alive.

Decision

Supervision: halt is a terminal state, not a crash.

  • Every pipeline child is restart: :temporary (lib/capstan/pipeline.ex child specs; the supervisor is :one_for_one). A fail-closed halt exits {:shutdown, {:halt, reason}} — not a crash — so a halted child is removed and NEVER restarted into a livelock, and the host supervisor (the consumer's tree) stays up. Restarting, reprovisioning, or paging is the host application's decision, informed by the [:capstan, :connection, :halt] / [:capstan, :assembler, :halt] telemetry (value-free, ADR-0007).
  • The Connection forwards frames to its receiver with a plain send/2 — a send to a dead receiver is a silent no-op, never a raise that could resurrect a halted pipeline. The corresponding silence hazard is closed by MONITORING the receiver: on its :DOWN the Connection stops fail-closed (:receiver_down) instead of streaming into a dead pid. Monitoring, not linking, so a :shutdown stop never couples the two children's restarts.

Liveness: two timers and two counters (lib/capstan/connection.ex).

  • SET @master_heartbeat_period asks the master for a HEARTBEAT_LOG_EVENT after heartbeat_period_ms of idleness — self-contained, independent of the server's slave_net_timeout — so even a quiet-but-healthy stream keeps delivering frames.
  • A parent-side liveness timer fires when NO frame (event or heartbeat) arrives within stream_timeout_ms (default 4× the heartbeat period: three missed beats): it emits [:capstan, :connection, :stream_timeout], kills the recv-blocked reader, and reconnects. Epoch-guarded — a stale timeout queued before a reset is ignored. Socket keepalive: true is the OS backstop for a fully-dead peer, deliberately not primary (~2 h default idle).
  • Two INDEPENDENT failure budgets: the command budget (max_command_retries, reset when a frame arrives — correct for transient pre-establish faults) and the established-then-dropped cycle counter (NEVER reset by frames — a duplicate server_id evicts this replica after each healthy-looking establish, and a frame-reset counter would livelock forever while emitting healthy telemetry; it halts :server_id_conflict instead). stream_timeout_ms must exceed heartbeat_period_ms or start-up refuses (:invalid_liveness_config) — a window at or below the interval would false-drop a HEALTHY idle stream; both constructors into the process enforce the identical defaults-then-compare predicate.

Consequences

  • A halted pipeline requires an operator (via the host app); the library never decides to restart past a data-integrity condition. This is the contract — availability posture is the consumer's layer.
  • A silent partition is bounded by stream_timeout_ms, made visible by telemetry, and a persistent one halts :stream_stalled once the cycle budget is spent.
  • Any future child keeps restart: :temporary and routes telemetry halts through Capstan.Telemetry; a restartable child is a design regression (the livelock class).