Failure and recovery

For operators who need to know what happens — and how long it takes — when a node, the leader, or the arbiter fails. Assumes the guarantees; leases covers the clock discipline underneath this page.

Fief has exactly three parties that can fail: a member node, the planner's leader seat, and the arbiter. Each failure has a different blast radius — a node failure costs its keys a bounded outage, a leader failure costs a pause in rebalancing and nothing else, an arbiter failure freezes the cluster's shape — and none of them, in any combination or order, produces two owners of one key. Partitions are the same three failures seen from one side; the netsplit matrix maps each partition to its row, and this page explains the machinery behind those rows.

Node failure

From the outside, a crashed node and a partitioned node are indistinguishable — so Fief never tries to distinguish them. Every recovery decision is lease-gated: nothing is reassigned because a TCP connection dropped or a nodedown fired; vnodes move only after the node's lease has expired as judged by the arbiter. Signals like nodedown prompt the planner to check the leases early — they accelerate the reaction, never authorize it.

The two sides of a node failure run independently, and their timing is engineered so the first strictly precedes the second:

The failing side fences itself. A node that cannot confirm lease renewal stops serving unilaterally at TTL − margin on its own clock — self-fencing, no permission required, no network needed. The fence is a kill, not a request: vnode agents and everything linked under them are terminated, key processes included (under the default on_fence: :terminate; the weaker modes leave survivor processes running and forfeit the side-effect guarantee they document). A Fief.Cache vnode's entries die with their table's owning process, unconditionally. After the fence, messages sent to the node vanish silently — no reply, no {:moved, ...} — and callers see timeouts.

The survivors take over, later. Once the arbiter observes the lease expiry at TTL, the planner reassigns the dead node's vnodes at new epochs. Vnodes the node plainly owned are assigned anywhere healthy with no donor — there is nothing safe to pull state from, so every key rebuilds from durable truth on its next touch, entering through init/3 with a :fresh source (the key lifecycle). Vnodes the dead node was receiving in an open transfer go back to their still-healthy donor, which resumes what it never shipped and discards the rest as tainted — the transfer page covers why.

What is lost is exactly what the no-delivery-guarantees posture already priced in: the node's in-memory key state and every mailbox on it. In-flight calls time out with unknown outcome; un-persisted state is gone. Durable truth is your job — a node death is the case that advice exists for.

Fail-fast: a sick node surfaces as its top process exiting

Everything above is what the survivors do. From inside the failing instance, Fief never tries to hide a sick node behind a silent internal reboot. The instance's own supervision tree is a thin outer supervisor over a kernel — Node, Planner, Vnode.Manager, Authority, Router — mounted as one rest_for_one unit with its own restart budget; by the time that kernel process exits, its local recovery is already exhausted, so the outer does not retry it. The kernel is significant, and the outer runs auto_shutdown: :any_significant, so a kernel exit brings the whole instance down and the outer supervisor itself exits with reason :shutdown. A Fief node going down always means its top process exits — the user's supervision tree decides recovery through the :permanent / :transient / :temporary child spec chosen for {Fief, opts}, and whatever it decides, the verified recovery path is always the one this page already describes: lease expiry, reassignment with no donor, cold rebuild on next touch.

This is also why a timed-out shutdown-leave is not a new failure mode. Its sentinel sits outside the kernel's restart unit and blocks the tree's fall for at most its configured leave_on_shutdown deadline; past that, the supervisor kills the sentinel and the tree falls exactly as it would on any other exit. The keys that never handed over land in precisely the state this section already covers — their in-memory state gone, rebuilt from durable truth on next touch — because a timed-out graceful leave and an ungraceful one converge on the same lease-gated takeover.

The failover floor

Key unavailability after a node death is bounded below by the lease TTL:

unavailability ≈ TTL  +  planner reaction  +  per-key rebuild on touch

The TTL term is a floor no mechanism can tunnel under — a faster signal (a nodedown, a health check, an operator's certainty) cannot safely shortcut it, because the "dead" node may merely be partitioned and still serving on the far side; the arbiter can only wait out the lease. The planner-reaction term is small (a nodedown hint usually makes it immediate; the poll cadence bounds it otherwise). Sizing the TTL is therefore the availability-versus-heartbeat-cost dial — tuning — with 3–5 s comfortable against a healthy Postgres.

The same arithmetic read from the other side: a node partitioned from the arbiter goes dark at TTL − margin even if the rest of the Elixir cluster can see it perfectly — the strict fence-before-reassignment order is what makes the takeover safe, and the gap between the two deadlines is a window where the keys are served by nobody. That window is the product working as designed.

Leader failure

Leadership gates exactly one thing: the planner's fenced writes. So a dead leader — or a leader partitioned from the arbiter, which is the same thing from the arbiter's side — costs a pause in rebalancing and failure reassignment, nothing else. Routing, key serving, leases, and self-fencing consult the leader for nothing and are unaffected. The cluster runs leaderless intervals all the time (every deploy that restarts the leader is one); a leader failure is not an incident tier, it is a scheduling gap.

Recovery is by term. Standbys observe the leader's absence (poll with jitter, or a nodedown hint) and campaign; one wins the next monotonic term and its planner reads membership, leases, and the vnode table and resumes. Open transfers resume exactly where the table says they are — every completed move was its own committed write, so a leader dying mid-rebalance loses nothing. A deposed leader that is somehow still running is harmless: every write it attempts carries its old term and is rejected by the term ratchet, and its next campaign finds the seat held at a higher term.

Because leadership needs none of the lease layer's clock conservatism (a wrong leader is safe, just idle — the term ratchet holds), election can run as hot as its mechanism allows. The place this matters is the compound failure — a member and the leader dying together — where the member's keys wait for a new leader to perform the reassignment; fast election trims that tail.

Arbiter failure

The arbiter is the one party whose failure stops the world from changing. With it unreachable, nothing that requires a write can happen — no joins, no leaves, no rebalancing, no failure reassignment — and what serving nodes do is governed by the on_arbiter_loss instance option:

• :freeze (the default). Nodes keep serving their last observed epoch while their leases remain live. An outage shorter than the fence window is effectively invisible: renewals resume, nobody fences, the planner picks up where it left off. An outage that outlasts the window drives every node to self-fence — the whole cluster goes unavailable — but the coordination state never moves, so it comes back consistent: when the arbiter returns, nodes re-acquire leases and rejoin against the same table.
• :serve_last_epoch. Nodes keep serving past lease expiry. This forfeits state and side-effect single-ownership outright — when the arbiter returns, the planner will reassign expired leases while the stale side may still be serving — and is sound only for workloads where that is acceptable by construction (read-mostly, externally-fenced writes). This paragraph is the caveat; the mode says it loudly in configuration too.

Either way, the failure mode is the accepted v1 trade stated in the overview: Postgres is a single point of failure for coordination. A healthy cluster keeps serving through a short arbiter outage; a long one takes availability, never consistency.

Verified by

Design notes: docs/design.md §6.4 (node failure and the clock discipline, model-checked in specs/FiefLease.tla), §6.5 (leader failure), §6.6 (the matrix); docs/implementation.md §2 (fencing as supervision kill). docs/leave-on-shutdown.md §2.6 is the decision record for the fail-fast split between the outer instance supervisor and the kernel. See verification for what model-checked means here.