Fencing modes

For application developers choosing on_fence: for a key module. Assumes the key lifecycle.

Fencing is what a node does to itself when it can no longer prove its ownership is current — lease renewal failing, arbiter unreachable. Per guarantee 3, it stops serving before any other node is allowed to take over. This page is about the one policy decision inside that mechanism: what happens to your key processes when their node fences.

The default answer is: they are killed. Merely muting a process is not fencing — a muted-but-alive process still fires timers and side effects, and "the new owner started" plus "the old process still acts" is precisely the double-ownership the guarantee exists to prevent. The kill is a supervision teardown whose completion time does not depend on user code: a slow callback cannot keep a process alive past the fence.

The mode is per key module, because guarantee needs differ per workload within one cluster — a session registry and a hand-rolled look-aside cache built on Fief.Key can share an instance with different modes. (If what you want is a coherent cache rather than a hand-rolled one, Fief.Cache needs no fencing decision at all — see below.)

use Fief.Key, on_fence: :terminate   # | :continue | :callback

Two things hold in every mode, because the framework enforces them independently of your choice:

• Routing single-ownership. A fenced node stops accepting key messages; cross-node message processing has exactly one owner throughout.
• The fence itself. The mode changes what survives the fence, never whether the fence happens or when.

:terminate — the default

Key processes are killed at the fence deadline, before the node's lease can expire from anyone else's point of view.

What it forfeits: nothing. This is the only mode under which guarantee 1 covers live processes and their side effects: at most one live process per key, anywhere, at any instant — so timers, external writes, and everything else your process does fire on at most one node.

Use it unless you can articulate exactly why the overlap the other modes permit is harmless for your workload. Un-persisted state dies with the process; that is the no-delivery-guarantees posture, not a property of this mode.

:continue — survive the fence, forfeit side-effect safety

Key processes survive the fence and keep running. Fief's routing to them stops (that much is unconditional), but the processes themselves live on: timers fire, scheduled work runs, anything holding their PID can still talk to them.

What it forfeits, loudly: guarantee 1 for state and side effects. After the lease dies, a new owner elsewhere starts serving the same keys while the old processes are still alive on the fenced node. For as long as that overlap lasts, two processes for one key exist in the cluster. If your key processes write anywhere or emit anything, :continue is the wrong mode.

Name the worst case precisely, because the open mailbox makes it sharper than "a stale process lingers": a :continue survivor that subscribed to PubSub in init keeps acting on every broadcast through its own handle_info/2 while its successor serves the same key on another node — a live split-brain actor, potentially for a long time (survivors are reaped only when this node next initializes that key's vnode; see the survivor lifecycle below). This combination — :continue plus side-effecting external subscriptions — is opt-in twice over, and it is the one to refuse: keys with external message sources that must not act after the fence belong in :callback, which exists precisely so they can unsubscribe or demote themselves to read-only.

What it's for: workloads where a stale survivor is harmless by construction — read-mostly caches where local code holding a PID reading stale data is acceptable, and the authoritative data lives elsewhere anyway. A stale-survivor Key-based cache like this is a different animal from Fief.Cache, which has no on_fence option and drops entries at any ownership change rather than letting them survive — readers who want a cache, not survivors, should start there instead.

The survivor lifecycle: :continue (and :callback) servers are supervised outside the vnode's kill tree from birth, in a per-instance limbo supervisor, which is what lets them outlive the fence — and Fief's own kernel restarts (a crashed coordination process fences the node, which is exactly when survivors must keep living). Survivors are enumerable (Fief.Key.Limbo.entries/1), and are reaped when the same node next initializes that key's vnode — so one node never hosts a survivor and a fresh serving process for the same key. Until that re-initialization, survivors linger (bounded cleanup on reconnect is a documented gap inside this mode's forfeit, not a violation of the default mode's guarantee). Two things do end a survivor early: instance shutdown, and — a deliberate, should-never-happen exception — a crash of the limbo's own bookkeeper, whose entry map is what makes survivors reapable; rather than risk unreapable survivors (a double-serve hole), the bookkeeper and its survivors share a fate (Fief.Key.Limbo records the decision). "Instance shutdown" means the supervision tree actually tearing down: by default a graceful shutdown now runs Fief.Node.leave/1 first, but that drain is scoped to ordinary vnodes — limbo is separate machinery the leave never touches, so survivors outlive it and die only once the tree itself finally falls. Survivor liveness is best-effort, crash-equivalent — the same posture as everything else in this mode's forfeit.

:callback — your process decides, under a deadline

At the fence, handle_fence/2 is fanned out to the key module's processes, carrying the advisory's deadline:

@impl true
def handle_fence(_deadline_ms, state) do
  MyPubSub.unsubscribe(state.topic)
  {:continue, demote_to_read_only(state)}   # or {:stop, :fenced}
end

{:stop, reason} stops the process; {:continue, state} keeps it alive with the same survivor semantics as :continue above. The callback runs under a deadline (deadline_ms, from the handle_fence_deadline option, default 50 ms — validated at instance start to fit inside the fence margin): the deadline is advisory, the fence is not. A callback that overruns doesn't delay the fence; don't put slow flushes here and count on them.

What it forfeits: guarantee 1 for state and side effects, at your process's discretion. Two workloads earn that discretion:

• Keys with external ingress that must go quiet at the fence. A key that subscribed to PubSub or armed timers in init unsubscribes / demotes itself to read-only in handle_fence and answers {:continue, state} — keeping its state warm for local readers without becoming the split-brain actor described under :continue. This is the sanctioned pairing for open-mailbox keys that want to survive.
• Keys whose writes are already fenced externally — every write carries a token some other system validates — so a lingering process is harmless and killing it would be pure availability loss.

If neither describes your keys, use :terminate.

Choosing

(The table is advice; the per-mode forfeit statements above are the contract.)

Verified by

• test/fief/key/vnode_impl_test.exs — describe "fence modes": all three modes at the fence — :terminate kills, :continue survives outside the kill tree, :callback receives the fan-out and both answers behave as stated.
• test/fief/key/limbo_test.exs — survivors outlive a Fief.Node crash, die with the instance, and die with the limbo bookkeeper's own crash (:one_for_all fate-coupling — the survivors-stay-reapable trade recorded in Fief.Key.Limbo).
• test/fief/node_test.exs — describe "self-fencing": the fence fires unilaterally at TTL − margin, before any peer may act.
• test/fief/keyed_netsplit_test.exs — describe "row 2: node↔arbiter partition": a fenced node's keys stop serving; takeover happens only after arbiter-judged expiry.

Design notes: docs/design.md §8 (the modes and their rationale), §6.4 (the fence timing); docs/implementation.md §2 ("fencing is a kill, not a request") and §7 (the limbo mechanism).