Netsplit matrix

For operators and users diagnosing partition behavior — "why is my key unavailable right now?" Assumes the guarantees.

Every network partition Fief can experience resolves to one of the rows below. The library's answer is always the same shape — downtime over inconsistency: a side that cannot prove its ownership is current stops serving before anyone else takes over, so keys go unavailable rather than doubly owned. This page is the map from "which link is broken" to "what a caller observes," and each row links the deterministic test that reproduces it. A behavior with a linked repro is a different class of claim than a behavior merely asserted.

Two guarantees frame every row, and their scope conditions travel with them: routing single-ownership is unconditional — no message is ever processed by a node that is not the key's current owner under a live lease, regardless of partition or fencing mode; state and side-effect single-ownership holds under on_fence: :terminate (the default) for Fief.Key, where the fenced side's key processes are killed. Under the weaker modes the fenced processes survive and that layer is forfeited — see Fencing modes. For Fief.Cache this layer is unconditional: entries live only in the table owned by the vnode's serving process and die with it, so there is no weaker mode to forfeit it under.

The matrix

Row 1 — nodes split from each other, arbiter reachable

Because ownership is derived from the arbiter's table and every side can still reach it, membership and epoch stay authoritative throughout: each vnode has exactly one owner. A caller that happens to sit on the owner's side of the split is served as if nothing happened. A caller on the far side has its delivery dropped by the broken link, and the request loop surfaces that as {:error, :timeout} (or a {:moved, ...} correction if it routed on a stale hint) — the owner never executes it and no second node executes it either, so routing single-ownership holds unconditionally.

The subtler case is a key mid-transfer when the split falls between the donor and the recipient. The recipient cannot pull the key's state from the donor, so it leaves the key unavailable rather than starting a second copy from scratch — a pull timeout never escheats while the donor might be alive. The key stays unavailable until the partition heals (the pull then completes) or the donor's lease dies. This is downtime-over-inconsistency at key granularity; the alternative — escheating on a mere timeout — has a model-checked counterexample.

Verified by test/fief/netsplit_test.exs — describe "row 1: node↔node partition, arbiter reachable" (the owner keeps serving, the partitioned sender times out, exactly one owner throughout) — and test/fief/keyed_netsplit_test.exs — describe "row 1: node↔node partition, arbiter reachable — escheat gating (⊨ rule 1)" (with the donor alive and unfenced, pulls go unanswered, the recipient's key stays unavailable, escheat never fires, and the heal completes the handover).

Design notes: docs/design.md §6.6 (row 1), §6.3 (escheat gating).

Row 2 — a node split from the arbiter

The isolated node's lease renewals fail. It does not wait to be told it is gone: it self-fences at TTL − margin, measured on its own monotonic clock from the renewal send time, unilaterally and with no communication. Peers may treat its lease as dead only at TTL on the arbiter's clock — so the fence strictly precedes any reassignment, and there is a window where the node has already stopped serving but the arbiter still counts it among live holders and nobody may take over yet.

What "stops serving" does is kill the vnode agent and everything it owns, at the fence deadline — key processes for Fief.Key, under the per-key on_fence policy (:terminate by default), and the entry table for Fief.Cache, unconditionally, with no policy to set. Sends arriving after that vanish silently. After the arbiter observes expiry at TTL, the planner reassigns the fenced node's vnodes with no donor to pull from, and the survivors rebuild those keys cold. The window between fence and reassignment is unavailability by design — the lease TTL is the failover floor, and no hint channel can beat it because a partition and a crash are indistinguishable from the arbiter's side.

Verified by test/fief/netsplit_test.exs — describe "row 2: node↔arbiter partition" (self-fence at TTL − margin with agents killed strictly before reassignment at TTL; peers proceed only after expiry) — and test/fief/keyed_netsplit_test.exs — describe "row 2: node↔arbiter partition" (the fence kill takes the key servers with the agents before the lease expires; survivors rebuild the keys cold after reassignment).

Design notes: docs/design.md §6.6 (row 2), §6.4 (self-fencing and the margin arithmetic, model-checked in specs/FiefLease.tla).

Row 3 — arbiter down for everyone

With the arbiter unreachable to all nodes, nothing that requires a write can happen: no joins, no leaves, no rebalancing, no failure reassignment. The planner freezes because it cannot CAS the table. What the serving nodes do is governed by on_arbiter_loss.

Under the default on_arbiter_loss: :freeze, nodes keep serving their last epoch while their leases remain live. An outage shorter than the fence window is effectively invisible: renewals resume when the arbiter returns, nobody fences, and the planner writes again on its normal cadence. An outage that outlasts the window drives every node to fence — the whole cluster becomes unavailable — but the coordination state never moves, so it becomes unavailable, never inconsistent. (The :serve_last_epoch alternative trades this: nodes keep serving past lease expiry, which forfeits single-ownership for keys with side effects and is sound only for read-mostly workloads. See the guarantees page and design notes.)

Verified by test/fief/netsplit_test.exs — describe "row 3: arbiter down (on_arbiter_loss)", which covers both the short outage (planner freezes and writes nothing, nodes serve their last epoch, full recovery when it returns) and the long outage (every node fences — unavailable, epoch unchanged, never inconsistent).

Design notes: docs/design.md §6.6 (row 3), §8 (on_arbiter_loss as the node-level member of the degradation-policy family).

Row 4 — leadership mechanism down, or the leader split from the arbiter

Leadership gates only the planner. If the leadership mechanism is unreachable, or the current leader is partitioned from the arbiter, there is simply no planner running: no rebalancing and no failure reassignment happen until leadership is re-established. This is identical to any ordinary leaderless interval — routing, leases, and self-fencing are all unaffected, because none of them consult the leader.

Recovery is by term. A standby campaigns and wins the next term, and its planner takes over reassignment. A deposed leader whose planner outlives its node's fence is harmless: every write it attempts is rejected as stale by the term ratchet, so it changes nothing, and its next campaign finds the seat held at a higher term and terminates it. The cost of a leader failure is a pause in rebalancing, nothing more.

Verified by test/fief/netsplit_test.exs — describe "row 4: leader↔arbiter partition" (planner frozen during the outage; takeover by term; a zombie leader's writes rejected as stale; the deposed leader terminated on recovery).

Design notes: docs/design.md §6.6 (row 4), §5.1 (leadership gates fenced writes only).

Row 5 — formation authority down

This row applies only to post-v1 ra deployments, where the StateStore is itself a replicated group and a separate Formation contract decides which nodes are authorized to constitute the quorum. Its behavior is fail-static: an unreachable formation authority freezes membership — no joins and no ejections — while a formed group keeps serving leases, membership, and the table.

In a v1 deployment this row does not apply. Postgres is the arbiter and plays all three authority roles; there is no separate formation authority to lose, and the contract is vacuous. There is accordingly no v1 test for this row.

Design notes: docs/design.md §6.6 (row 5), Appendix B (Formation and the ra path).

Row 6 — presence channel severed

Presence is the push channel that tells nodes about membership and ownership changes promptly. It is a hint accelerator, never an authority: losing it costs latency, not correctness. With presence severed, nodes fall back to their poll timer for the same information. Everything still converges — a joiner's transfers still open, donors still observe the change, drain, and settle — just at poll tempo instead of on a push, and no caller observes a correctness difference. Every hint channel in the system can be severed and the answers stay correct, only slower.

Verified by test/fief/netsplit_test.exs — describe "row 6: presence severed", which drives a full join-and-transfer with zero hints and zero explicit refreshes and confirms everything converges on the poll seam alone.

Design notes: docs/design.md §6.6 (row 6), §4 (hints accelerate, never authorize).