The transfer protocol

For users and operators who want to know what happens — and what callers observe — when key ownership moves between nodes. Assumes the guarantees; the key lifecycle covers the same window from inside your callbacks.

Ownership moves whenever the cluster rebalances: a node joins and must take its share, a node leaves and must give its share up, a node dies and its share is reassigned. This page walks the graceful case — both nodes alive, state handed over rather than lost — and then the rules that govern every way it can be interrupted. Recovery from death without a handover is the failure page's subject.

The protocol is model-checked (specs/FiefTransfer.tla): each rule below marked model-checked exists because the model checker produced a concrete double-ownership or stale-state counterexample without it. This page states each rule by its observable consequence; the derivations live in verification.

One scope note: incremental state handover is a Fief.Key behavior. Fief.Cache deliberately opts out — on any ownership move its entries are dropped and the new owner starts cold, so for a cache instance this page reduces to its first section (the flip) and the caller experience is misses, never staleness.

Authority moves instantly; state follows

The core shape: the vnode-level ownership flip is one compare-and-swap in the arbiter, and the new owner is authoritative for the whole vnode from that instant — but each key's state follows in its own per-key handover. The unit of atomicity is the key, not the vnode: a vnode may hold thousands of key processes, and draining them all before anyone could route again would stall every key behind the slowest extraction. Instead, no key is ever blocked by another key's handover.

A transfer runs in five overlapping phases:

1. The flip. The planner writes {owner: B, prev_owner: A} to the vnode's row; the epoch bumps. Hints go out, but hints only accelerate — each node acts when it observes the row. B is authoritative for the entire vnode immediately; the transfer stays open — prev_owner set — until phase 5.

2. Freeze. A observes the flip (by hint, by poll, or by B's first pull arriving) and freezes the vnode. Each live key process gets the two-phase freeze treatment: the handle_freeze advisory, then extract_state/1 in mailbox order, with the extract_deadline timer (default 5 s) as the backstop for keys that are flooded or stuck: past the deadline the still-pending key is killed outright rather than extracted, so it produces no residual and escheats — B rebuilds just that key from durable truth on first touch, the same as any key A never held, not the whole vnode. Extraction is the process's last act for every other key — the old process stops inside the handover, before the new one starts, which is how exactly-one-live-process-per-key holds throughout. The extracted blobs become residuals: inert state held for handover, running no timers and producing no side effects. From the freeze on, anything sent to A directly is answered {:moved, epoch, delta}; the caller's node patches its routing and retries at B, invisibly to the caller.

3. Pull on first touch. When B receives the first message for key K, it pulls K from A; A hands the residual over (a grant), and K's next incarnation starts from it on B via init(key, {:residual, blob}, ctx). Messages arriving for K while the pull is in flight are pended, bounded by max_pending_per_key (default 128; overflow surfaces as caller timeouts, never unbounded memory). The cost profile: a hot key pays one extra round-trip on its first post-flip message; a cold key blocks nothing and nobody.

4. The sweep. In parallel, A pushes its remaining residuals to B in the background — sweep_rate keys per tick (default 100), one tick per sweep_interval (default 1 s) — so cold keys migrate without waiting to be touched. Every residual, granted or pushed, is retained by A until B acknowledges it. The sweep rate is deliberately the pacing knob: it bounds per-node transfer bandwidth, and for a leaving node it is the drain-time knob (join and leave).

5. Settle. When A's residual ledger is empty — everything granted, pushed-and-acked, or never present — A reports "I hold nothing," and only on that report does the planner clear prev_owner. The donor is the one party that authoritatively knows it is empty; settling on any other evidence lets an oblivious donor keep answering for keys after the row says the transfer never happened. A transfer being open is defined as prev_owner being set: that fact lives in the arbiter's table, one SELECT away, and survives leader failover — a new planner resumes exactly the open transfers the row says are open.

Two per-key epilogues from the key lifecycle apply here: a residual is used at most once — if the incarnation it started dies, recovery goes back through durable truth, never through the stale blob — and a residual whose blob exceeds max_blob_size (default 64 KB) is dropped at freeze and the key rebuilds from durable truth on B, so un-persisted state in an oversized blob is lost.

The rules when it goes wrong

Everything above assumed both parties stay alive and connected. The rules below govern the interruptions; each is model-checked, and each trades availability — never consistency — when in doubt.

A pull timeout never rebuilds the key

If B's pull goes unanswered — A is on the far side of a partition, or just slow — B retries on pull_retry_interval (default 1 s) and the key stays unavailable. It does not rebuild the key from durable truth, however long the silence: A may be alive and unfenced over there, still holding the live process, and rebuilding would put two live processes on one key. B rebuilds from durable truth (escheat — the key reverts to the system and is re-granted) only on positive evidence: A explicitly answers not here for this key in this session, or the session itself ends because A's lease died or the planner aborted the transfer. Silence authorizes nothing. This is downtime over inconsistency at key granularity, and it is the row-1 behavior on the netsplit matrix: the key waits until the partition heals or the donor's lease dies.

Stale handovers cannot land

Every grant and push carries its transfer session, and a blob from any other session — a push surviving from an ancient custody period, delayed in flight — is dropped before your code ever sees it, without acknowledgment (an ack means "delivered" and would release the donor's retained copy; acks are session-scoped for the same reason). B also injects each key at most once per session, even if the injected process dies: once injected, the shipped blob is stale by construction — the process may have written and persisted newer state — so a late in-session duplicate is acknowledged without injecting, and a dead injected key recovers through durable truth. The consequence you can rely on: an old copy of a key's state can never resurrect, no matter how messages are delayed, duplicated, or reordered.

The recipient dies: back to the donor, shipped state is tainted

If B dies mid-transfer while A is healthy, the planner reassigns the vnode back to A — never to a third node. Reassigning elsewhere would erase from the row the one healthy node that may still believe it owns the vnode. On regaining ownership, A resumes only the residuals it never shipped (as fresh local incarnations, origin: :resume); every residual that was shipped — acked or not — is tainted and discarded, because B may have mutated and even persisted newer state before dying, and blobs carry no version to compare. A tainted key rebuilds from durable truth on its next touch. Trust durable truth over any in-memory copy of unknowable age.

The donor dies: the session is aborted, keys rebuild from truth

If A's lease expires mid-transfer, the planner aborts the session — settles the row without a report — and B, now with positive evidence the session is over, rebuilds the unmigrated keys from durable truth as they are touched. Their in-memory state died with A; this is the same loss class as any node death, and the same advice applies — durable truth is your job.

Nothing ever pulls from a stale hop

State is only ever pulled from the immediate prev_owner, and the planner never moves a vnode whose transfer is still open (max_concurrent_transfers bounds these open rows — it is a cap on in-flight pull traffic, not on CAS throughput). If owner and donor are both dead, the vnode is reassigned with no donor at all and every key rebuilds from durable truth. The hazard this forecloses: keys already migrated to B were mutated there, while A still holds their pre-transfer state — a successor pulling from A would silently roll those keys back. Rebuilding from durable truth is always safe; pulling from a stale hop is not, so it cannot happen.

What callers observe, phase by phase

The per-sender ordering caveat from the guarantees applies in the flip window: messages from one sender may be split across the old and new owner without an ordering guarantee across the pair — exactly-one-owner-at-a-time holds throughout, ordering across the handover does not.

Verified by

Design notes: docs/design.md §6.3 is the normative protocol statement, with each model-checked rule's rationale; docs/implementation.md §7 maps them to code; the Fief.Transfer moduledoc carries the machine/embedding split. The model itself is specs/FiefTransfer.tla — see verification.