The key lifecycle

For application developers writing use Fief.Key modules. Assumes getting started; the concepts pages deepen it but are not prerequisites.

A key is a GenServer that Fief places, moves, and fences. Its process is born on the node that owns the key, serves messages there, and ends when ownership moves, the node dies, or the process stops itself. The whole lifecycle in one line:

init (every incarnation: fresh, escheat, transfer, resume)
   → serve (handle_message — routed, validated | handle_info — yours, unvalidated)
   → handle_freeze advisory → extract_state + retire | {:stop, …} | deadline kill
   | die-with-node

One way in, three ways out. This page walks the spine — one entrypoint, two mailboxes, three exits — then the caveats that earn their place: key hashing, and what callers observe while a key is mid-move.

One way in: init/3

init(key, source, ctx) runs at the start of every incarnation of a key process. You never start key processes yourself — routing by key is the whole interface, and the process starts on demand, on the owning node. Return {:ok, state} to serve, or {:stop, reason} to refuse (the caller gets an error; the next message tries again).

source says what this incarnation starts from; ctx.origin says why it exists:

Two rules govern the whole table:

• Process-scoped wiring is re-established here, every time. Timers, subscriptions, monitors — anything keyed to self() — died with the previous incarnation, and nothing rides the blob except what extract_state/1 put there. init is where you re-arm them; because init runs on every incarnation, arming them there is automatically correct wherever and however the key materializes.
• ctx.origin is informational, never procedural. Correct code may ignore it entirely; branch on it for logging, metrics, or cache invalidation, never for whether to consult durable storage.

:fresh — locate durable truth, or create it

:fresh does not mean "this key never existed." It runs on the first touch of this incarnation: after a key process crashes (crashed key processes are deliberately not restarted by supervision — the next message cold-starts a fresh one), after a {:stop, ...} retirement, and after a node death erases a key's process entirely. A fresh owner after a donor's death cannot distinguish "this key existed elsewhere before" from "this key never existed" — there is no transfer session, no residual ledger, no memory, just a message arriving for a key with no process. So there is exactly one correct reading of :fresh: locate this key's durable truth, or create it.

origin: :escheat narrows that only informationally: the protocol has positive knowledge that a live copy recently existed and its in-memory state was lost. (The name is the feudal one — the key reverts to the system and is re-granted.) It is the hook for anything extra you want to do when you know recovery is happening — metrics, invalidating leases you hold elsewhere, conservative re-reads — but the state-building path must be the same one :first_touch takes, because after a node crash surviving keys re-enter through :first_touch on their new owners.

If your keys have no durable truth — a pure in-memory accumulator — starting fresh and empty is correct and honest: the state is gone, and that is exactly the no-delivery-guarantees posture.

{:residual, blob} — a planned handover's product

When the cluster rebalances — a node joins, a node leaves gracefully — vnode ownership moves before key state does, and each key's state follows in its own handover: the old owner runs extract_state/1, the resulting blob ships, and the new owner's incarnation starts from {:residual, blob} with origin: :transfer. The blob is opaque to Fief; the only contract is that it round-trips through your own callbacks. Hot keys are pulled on first touch (costing one extra round-trip); cold keys migrate in the background sweep without blocking anyone.

A residual is used at most once per key per transfer. If the incarnation it started later dies, the shipped blob is not reused — it is stale by construction, because the process may have written newer durable state after arriving. Recovery goes back through :fresh, from durable truth.

origin: :resume is the local variant: a freeze was aborted (the donor regained ownership mid-transfer), so the extracted blob never shipped and instead resumes a fresh incarnation on the same node. Same shape, same handling — which is the point of the single entrypoint.

Serving: two mailboxes

A key process owns its mailbox the way any GenServer does. The framework claims its own reserved messages — every tag prefixed fief_key_ is reserved — and routes everything else to you. That splits serving into two paths with deliberately different guarantees.

handle_message/3 — routed, validated

handle_message(msg, from, state) serves one routed message at a time:

• {:reply, reply, state} answers the caller.
• {:noreply, state} leaves the caller waiting — reply later from any process (or a later incarnation) via Fief.Key.reply(from, reply), or let the caller time out. from is nil for casts.
• {:stop, reason, state} retires the process after this message; the next message cold-starts through init/3.

Messages arrive via MyKey.call(instance, key, msg) / MyKey.cast/3 (or Fief.Key.call/5 / Fief.Key.cast/4). Calls run in the caller and return {:ok, reply} | {:error, reason}; the error cases and what they mean for retries are on guarantees and delivery and errors.

Routed messages are delivered only while this node owns the vnode under a live lease — that validation is the framework's half of the contract, and handle_message is the only callback that gets it.

handle_info/2 — yours, unvalidated

Everything else in the mailbox — your timers, PubSub broadcasts, monitor :DOWNs, anything sent to a captured self() — is delivered to your optional handle_info/2 ({:noreply, state} or {:stop, reason, state}). A module that doesn't export it drops strays silently.

handle_info is not ownership-validated. A timer or broadcast can fire at any moment your process is alive, including while ownership is moving or gone (bounded by the fence kill under the default fencing mode). Mutate state from handle_info only in ways that tolerate that — or route the event through Fief.Key.cast/4 instead, which is ownership-validated, survives migration transparently, and is also the only ingress that reaches a non-resident key (a cast cold-starts the key on touch; a subscription only fires while you happen to be resident).

What the open mailbox is for, idiomatically:

• Write-behind persistence. Durable truth is your job, and write-through on every message is often too expensive; a timer armed in init flushing dirty state every few seconds is the standard economical answer.
• Self-eviction. An idle key arms its own timer in init and returns {:stop, :normal, state} when it fires; there is no built-in idle_timeout because you can now express it directly.
• React-while-resident. Subscribe to PubSub in init, act on broadcasts while resident, and catch up from durable truth in the next init — a legitimate pattern as long as missing broadcasts while non-resident is acceptable. (If it isn't, the events should be casts.)

Putting the spine together — one entrypoint re-arming its wiring, a validated routed path, and a write-behind flush:

defmodule MyApp.Tally do
  use Fief.Key   # on_fence: :terminate is the default

  @flush_every 5_000

  @impl true
  # Every incarnation starts here. :fresh means locate durable truth or
  # create it; the flush timer is process-scoped, so it is re-armed here,
  # every time.
  def init(key, :fresh, _ctx) do
    Process.send_after(self(), :flush, @flush_every)
    {:ok, %{key: key, count: MyApp.Store.get(key, 0), dirty: false}}
  end

  # A planned handover handed the previous incarnation's state back; the
  # store may lag it, so start from the blob — but the timer still needs
  # re-arming, exactly as above.
  def init(_key, {:residual, tally}, _ctx) do
    Process.send_after(self(), :flush, @flush_every)
    {:ok, tally}
  end

  @impl true
  # Routed and ownership-validated.
  def handle_message(:bump, _from, tally) do
    tally = %{tally | count: tally.count + 1, dirty: true}
    {:reply, tally.count, tally}
  end

  @impl true
  # Your own mailbox — NOT ownership-validated. Write-behind: one durable
  # write per window instead of one per message.
  def handle_info(:flush, tally) do
    if tally.dirty, do: MyApp.Store.put(tally.key, tally.count)
    Process.send_after(self(), :flush, @flush_every)
    {:noreply, %{tally | dirty: false}}
  end

  @impl true
  def extract_state(tally), do: {:ok, tally}
end

(This module is executed by test/guides/key_lifecycle_test.exs.) Note what the flush tolerates: under :terminate, the worst case is one stale write racing the successor's inside the fence window — acceptable for a tally. If a stale overwrite is not acceptable for your data, make the write conditional (compare-and-swap on a version) or route the event through cast/4 so it only runs under validated ownership.

One thing the open mailbox is not for: identity. Do not register a key process under a global name (:global, a cluster-wide Registry) — that reintroduces exactly the split-brain this library exists to prevent. The mailbox is open for ingress; the only address of a key is {key_module, key} through the routed API.

Three ways out

A planned handover: freeze

When ownership moves gracefully, the old owner freezes the vnode. For each live key, three things happen, in order:

1. The advisory. If you export handle_freeze(deadline_ms, state), it is delivered as a priority message (EEP 76 — the reason for the OTP 28 floor), so it arrives ahead of any backlog: you learn immediately that a freeze started and a deadline is running. It is pure information — the policy is yours. {:noreply, state} keeps draining (the idiomatic response under load flips a flag and has subsequent handle_message calls cheaply bundle {msg, from} pairs into state, to ride the blob and be served by the next incarnation); {:extract, state} ships this state immediately, abandoning the backlog (callers time out); and {:stop, reason, state} dies with no residual at all — the next incarnation starts :fresh, for when durable truth is already current and the cheapest transfer is no transfer.
2. The extract. extract_state/1 is requested in normal mailbox order — a healthy key quiesces exactly as you'd hope: in-flight validated callers get served first, nobody times out. Extraction is the process's last act; the old process stops inside the handover, before the new one starts, which is how exactly-one-live-process-per-key holds through a transfer.
3. The deadline. A donor-side timer (extract_deadline, default 5 s — per-node tuning) backstops the whole phase: a key that hasn't extracted when it expires is killed and converges through the died-before-extract path — no residual; the new owner rebuilds from durable truth via origin: :escheat. The deadline is what bounds a transfer against a flooded mailbox, and it is also the only mechanism that handles a key stuck inside a callback, where no message priority can help.

After the advisory, this incarnation is over — extract, your own return, or the deadline kill; there is no path back to serving. The advisory promises "you will stop," not "you will move": if the transfer aborts, your blob resumes a fresh local incarnation with origin: :resume.

If you bundle at freeze, max_blob_size is your budget: an over-cap residual is dropped at ship and the next incarnation escheats — the bundle is lost, strictly worse than having drained. Bound your buffer. (from is location-transparent — Fief.Key.reply/2 works cross-node from the next incarnation — but callers' timeouts keep ticking while their messages ride the blob.)

Retirement: {:stop, reason, state}

Retirement is unilateral: returning {:stop, reason, state} from any callback ends the incarnation, the framework notices via :DOWN, and the next routed message cold-starts through init/3 (:fresh, :first_touch). Messages already in your mailbox die with you and those callers time out — identical to the crash path, per no delivery guarantees. There is deliberately no retire-handshake with the framework: the liveness map forgets a pid only on :DOWN, which is exactly what makes exactly-one-live-process-per-key hold. This is also the self-eviction mechanism — timer in init, stop when it fires.

Die with the node

Crashes and fencing run no callbacks (under the default fencing mode; the other modes differ). The process is simply gone, along with its mailbox — this is why extract_state must never be your only persistence mechanism.

Keys are binaries

The key you pass to call/cast must be a binary, unless the instance configures a custom hasher — a module implementing Fief.Hasher (hash_key(term) :: non_neg_integer). The default hasher (Fief.Hasher.Default, :erlang.phash2/2) accepts binaries only; a non-binary key raises unless you either encode keys to binaries yourself or configure a :hasher that accepts the term. Hashers are modules, never funs, because the hasher module is the cluster-wide compatibility fingerprint.

A custom hasher receives the raw key term, so key identity becomes whatever the hasher makes of the term's structure — 1 and 1.0 are different terms, a charlist and a binary are different terms, and a struct key changes shape when your app adds a field (a rolling-deploy split risk). Encode injectively and version the hasher (a new module) deliberately; the fingerprint pins the module across nodes, but behavioral stability over time is your contract.

The key→vnode mapping is immutable for the life of the instance: the hasher module. Nodes disagreeing on where a key hashes is the double-ownership hazard at the key layer, so the hasher is validated against the authority's record when a node joins — a mismatch is fatal at startup, before the node touches shared state. Changing it means standing up a new instance and migrating (the same blue/green story as repartitioning — see instances). This hashing machinery is the shared Fief.Hasher substrate — Fief.Cache uses it identically, fingerprinting the same hasher module at join.

Key identity is {key_module, key}: multiple use Fief.Key modules coexist in one instance and share the vnode space, and the same key string in two modules names two different keys. The hash covers the key alone — the module is identity, not placement. Fief.Cache deliberately forecloses this multiplicity: one anonymous cache per instance, with no module identity to key on — see Using Fief.Cache.

What callers see mid-transfer

While a key's vnode is between owners:

• The first message for a hot key pays one extra round-trip (the new owner pulls the key's state from the old owner on first touch).
• Messages arriving during the pull are pended, bounded by max_pending_per_key (default 128); overflow surfaces to callers as timeouts rather than unbounded memory.
• The freeze phase itself is bounded by extract_deadline: one key that cannot reach its extract — flooded, or stuck in a slow callback — is killed at the deadline and rebuilt from durable truth on the new owner, rather than holding the transfer window open for everyone.
• If the old owner is unreachable across a partition, the key is unavailable by design — a pull timeout never triggers rebuild-from-truth, because the old owner may still be alive and serving on the far side. The key waits until the partition heals or the old owner's lease dies. This is downtime over inconsistency applied per key; the model-checked rules behind it are on the transfer protocol page.

One sizing caveat: handover blobs ship in-band with a hard cap (max_blob_size, default 64 KB — the failure log line names the option). A key whose extracted state exceeds the cap is not shipped: it is dropped from the handover and escheats on the new owner — rebuilt from durable truth — so un-persisted state in an oversized blob is lost, the same class of loss as a node death, and the same durable-truth advice applies. A freeze-time bundle counts against the same cap.

Verified by

Design notes: docs/key-redesign.md is the decision record for this contract (the single entrypoint, the open mailbox, the two-phase freeze); docs/design.md §6.3, §8; docs/implementation.md §7. The transfer rules are model-checked (specs/FiefTransfer.tla); see verification.