Leases

For users and operators reasoning about failover timing and the clock assumptions Fief makes. Assumes guarantees.

Every member node holds a TTL lease in the authority and must keep renewing it. The lease is the cluster's authoritative liveness record: serving vnodes is conditioned on holding a live lease, and every recovery decision — "this node is dead, reassign its vnodes" — is gated on a lease expiring, never on a network signal. This page explains the timing: who measures what, on which clock, and why the lease TTL is a floor no configuration can tunnel under.

Self-fencing: the node judges itself first

A node that cannot confirm renewal stops serving itself, unilaterally, before its TTL — no network round-trip, no reads, no permission. That is the mechanism behind downtime over inconsistency: by the time any peer is allowed to treat the lease as dead, the holder has already stopped. What "stops serving" does to your key processes is per-key policy, for Fief.Key — killed, by default (fencing modes). For Fief.Cache there is no policy to set: entries live only in the fenced process's table and die with it, unconditionally.

The two sides deliberately measure on different clocks and different deadlines:

• The holder measures on its own monotonic clock, from the moment its last confirmed renewal was sent (not acknowledged — the request may have committed at the arbiter even if the ack was lost), and fences at TTL − margin.
• Peers (the planner) treat the lease as dead only at TTL as judged by the arbiter's clock — expiry is observed in the authority, never inferred locally.

No wall-clock agreement between nodes is ever assumed; each party consults only a clock it owns, and the asymmetric margin is what makes the two judgments safe against each other.

The margin

The margin budgets the renewal round-trip plus twice the clock-drift bound — twice, not once, because drift can swing from one extreme to the other across the measurement interval. This is a model-checked rule: the 1× margin has a machine-found counterexample. You normally don't set it — margin defaults to :auto, computed from the renewal cadence and drift bound.

Renewal failures are two different events:

• :unreachable (the arbiter can't be reached) keeps the fence timer running and keeps retrying at the renewal cadence — the fence clock is already ticking, so there is nothing more conservative to do.
• :expired (the arbiter answered: this lease is gone) fences immediately.

So a node partitioned from the arbiter goes dark before its TTL even if the Elixir cluster looks perfectly healthy from where it stands — both sides act only on what the arbiter says about themselves.

TTL is your failover floor

From the outside, a crash and a partition are indistinguishable, so recovery is lease-gated: the planner observes the expiry in the authority, then reassigns the dead node's vnodes. The unavailability floor for the affected keys is therefore

lease TTL + one planner reaction

and no hint channel can beat it. BEAM nodedown signals, presence events, and planner pings make the planner check the lease sooner; none of them authorizes acting before arbiter-judged expiry — a "down" signal may just be a partition, with the node alive and serving on the far side. Hints accelerate, never authorize.

Consequently lease_ttl is the tuning knob for failover time, bounded only by arbiter health and heartbeat cost (one small UPDATE per node per renewal interval). Against a healthy Postgres, a 3–5 s TTL with ~1 s renewal cadence is comfortable (advice — see guides/operations/tuning.md when sizing for production). None of this conservatism applies to leader election, which gates only fenced writes and may run as hot as its mechanism allows — but fast election can't beat the lease floor either; it only helps compound failures.

What this means for you

• Plan for affected keys to be unavailable for roughly lease_ttl plus a planner reaction after a node dies. If that number is too slow, lower lease_ttl — don't look for a signal-based shortcut, because there isn't one.
• A node cut off from Postgres fences even when the BEAM mesh is healthy. Arbiter reachability is part of your availability story; treat it that way operationally.
• Un-renewed doesn't mean un-served elsewhere: until the arbiter-judged TTL passes, nobody takes over — you get a gap, never an overlap. The gap is the product working as designed.

Verified by

• test/fief/node_test.exs — describe "lease renewal" (send-time basis, :unreachable retry posture), describe "self-fencing" (the fence fires unilaterally at TTL − margin, :expired fences immediately), and describe "margin arithmetic" (the :auto margin arithmetic, including the 2×-drift budget).
• test/fief/netsplit_test.exs — describe "row 2: node↔arbiter partition": the partitioned node fences before any peer may act; takeover only after arbiter-judged expiry.

Design notes: docs/design.md §4 (leases with self-fencing), §6.4 (clock discipline), §9 (TTL tuning). The margin arithmetic is model-checked in specs/FiefLease.tla.