Join and leave

For operators running cluster membership changes — deploys, scale-ups, drains. Assumes the guarantees; the transfers both protocols ride on are their own page.

Membership changes are the routine protocols: every deploy is a leave and a join. Both are built so the risky step cannot come first — a joiner is proven compatible and reachable before it can own anything, and a leaver keeps answering for its keys until every one of them has demonstrably landed somewhere else.

Join: prove yourself first, own things later

A node joins in two phases, and the order is load-bearing.

Phase one — connectivity and compatibility, zero ownership. The joiner reads membership from the authority, dist-connects to every live member and verifies the mesh, and validates its configuration against the namespace's authoritative record: partition count, protocol version, and the vnode impl's compatibility fingerprint (for Fief.Key and Fief.Cache, the hasher module — configuration marks which options are validated this way). The record is first-writer-wins: the first node to ever join a namespace writes it, and every later joiner is compared against it. Any mismatch is fatal at startup, before the node touches shared state — the error names the offending field, and the blast radius is one node that refuses to boot, not a cluster half-serving two incompatible layouts. Only after all that does the node register, acquire its lease, and appear with status joining — a full member of the mesh that owns nothing and receives no traffic.

Phase two — ownership arrives incrementally. The planner marks the node active and starts assigning it vnodes, one compare-and-swap per move, each move a standard transfer: the joiner takes only the over-capacity vnodes (what a balanced spread says it is owed), paced by max_concurrent_transfers. For Fief.Key each vnode's keys follow with their state — hot keys pulled on first touch, cold keys swept in the background; Fief.Cache has no state to follow, so its vnodes simply start cold on the joiner. Nothing about a join is special to the transfer machinery either way; a join is just N ordinary transfers with the same recipient.

Why phase one must complete first is mundane and absolute: the moment a vnode maps to the new node, every peer starts sending it traffic. A missing dist connection at that moment means timeouts and retry storms; an incompatible hasher means two nodes computing different homes for the same key — the double-ownership hazard at the key layer. Both are foreclosed before the first vnode can flip.

Leave: linger until everything has landed

A graceful leave is the mirror image, initiated explicitly: Fief.Node.leave(instance). The node sets its status to leaving; the planner observes and begins moving all of its vnodes to survivors — again, standard transfers. For Fief.Key that means every live key runs extract_state/1 and hands its state over rather than losing it (the freeze, from your callbacks' side); Fief.Cache has nothing to hand over by design — its vnodes report drained on the spot and the new owner starts cold (the transfer page covers why).

The leaver releases its lease and stops only when it appears as neither owner nor prev_owner for any vnode — that is, when every one of its transfers has settled; it re-checks on leave_drain_interval (default 100 ms). Because state moves lazily, the drain is the part that takes time, and the knob that governs it is the sweep rate, not the assignment rate: the vnode flips are cheap arbiter writes, but the residuals drain at sweep_rate keys per sweep_interval tick, each retained until acknowledged.

Shutdown runs this for you now, by default. Tearing an instance's supervision tree down — System.stop/0, SIGTERM, application shutdown, or a parent terminating the instance's own supervisor — triggers a shutdown-leave sentinel that calls Fief.Node.leave/1 and blocks the tree's fall until the node drains to :stopped or a deadline expires: the leave_on_shutdown instance option, default 20 s (configuration covers it, including sizing it against your orchestrator's grace period). Calling leave/1 yourself ahead of shutdown still works exactly as described above, and composes for free with the sentinel: it finds the node already :stopped and returns at once.

Opting out is the deliberate act, with leave_on_shutdown: false. Two cases warrant it: a deploy pipeline that already orchestrates its own drain ahead of SIGTERM, where the automatic leave would just be a redundant idempotent call; or an instance whose state is large enough that the leave-then-rejoin round trip's bandwidth cost outweighs a bounded unavailability window on restart. Either way, the failure mode you are opting into already exists and degrades safely: a leave that times out — on shutdown or from an explicit call — is just a node failure, and its un-handed-over keys rebuild from durable truth after lease expiry (failure) — an impatient shutdown costs cold starts, never consistency.

What callers observe during either protocol is exactly what they observe during any transfer — the phase-by-phase table on the transfer page — since that is all a join or leave is made of.

Verified by

Design notes: docs/design.md §6.1–6.2; docs/implementation.md §4 (Fief.Node's join/leave state machine) and §3 (ensure_config/2, the join-validation primitive); docs/leave-on-shutdown.md is the decision record for the shutdown-leave sentinel.