Runbook

Copy Markdown View Source

For operators running Fief in production: what a deploy looks like, what a failing node looks like, and the SELECTs that answer "what is my cluster doing right now?" Assumes the guarantees and failure and recovery; the schema queried here is the Postgres adapter's five tables.

A graceful deploy: leave on shutdown

Every deploy is a leave, automatically. When a Fief instance's supervision tree is torn down — SIGTERM, System.stop/0, application shutdown, or a parent supervisor stopping the {Fief, opts} child — a sentinel process runs a graceful Fief.Node.leave/1 and blocks the tree's fall until the node has handed off everything it owns, or a deadline expires. This is leave_on_shutdown, on by default at 20_000 ms. You do not have to orchestrate a drain yourself for the ordinary rolling-restart case; a plain kubectl rollout restart, or any orchestrator that sends SIGTERM and waits, already gets the graceful path.

What this looks like from the outside:

  1. The instance receives the shutdown signal. The node's status flips to leaving in the authority (visible in fief_members, below) while the rest of the instance — routing, key serving, the planner — stays fully up behind it.
  2. The planner observes the leaving status and moves the node's vnodes to survivors as ordinary transfers; keys hand off their state rather than losing it. This is the same mechanism a manual leave or a scale-down uses — a deploy is not a special case of it.
  3. Once the node owns nothing anywhere (neither owner nor prev_owner in fief_table), it releases its lease, publishes :stopped, and the rest of the supervision tree falls normally. From the outside: the process exits cleanly, and the cluster never had to wait out a lease expiry to notice it was gone.

The default, 20_000 ms, is deliberately under Kubernetes's default terminationGracePeriodSeconds of 30, so the drain gets a real chance to finish before SIGKILL lands. Keep terminationGracePeriodSeconds larger than leave_on_shutdown with real headroom — if they're equal, ordinary scheduling jitter between "SIGTERM sent" and "drain starts" can eat the margin and turn a clean leave into a killed one. If your nodes typically carry more state than the default window can drain, see tuning's transfer pacing before raising the timeout — sweep_rate is usually the more effective knob.

A timed-out drain is not a new failure mode: the supervisor kills the sentinel, the tree falls, and the cluster sees an ordinary node failure — the node's un-handed-over keys rebuild from durable truth after the usual lease-gated takeover. An impatient shutdown costs cold starts, never consistency.

Opting out. Set leave_on_shutdown: false if your deploy pipeline already orchestrates an explicit drain before sending SIGTERM (calling Fief.Node.leave/1 yourself and waiting for it), or if your workload's state is large enough that the leave-then-rejoin round trip's bandwidth cost outweighs the availability win. The full option — value type, default, and the shape gate that decides whether the sentinel exists at all — is documented in configuration. This behavior needs your Ecto repo to still be running when the drain executes — see the Postgres adapter's start-order note.

A sick node becomes visibly dead

Fief does not hide a persistently unhealthy instance behind a silent internal reboot. If the instance's internal machinery (the node agent, the planner, the vnode manager — everything under the hood) exhausts its own local recovery, the whole instance's top-level process exits rather than being quietly restarted in place. Concretely: the internal supervisor governing all of that machinery is mounted as a significant child of the instance's root supervisor, so when it gives up, the root supervisor exits too, with reason :shutdown.

What you observe: instead of an internal crash-loop generating an endless stream of lease/membership churn that the rest of the cluster has to keep reacting to, the instance disappears cleanly and immediately — one visible exit, not a flapping process silently retrying underneath you. Recovery decisions are yours: this is by design, because your own supervision tree already owns restart policy, backoff, and escalation for everything else in your application, and a Fief node repeatedly failing to stay up is exactly the kind of condition your ops tooling should see and alert on rather than have masked.

This has one concrete consequence for how you mount {Fief, opts}:

  • A :permanent child spec restarts the instance fresh after this exit — new registration, a fresh lease, a clean tracker — which is usually what you want.
  • A :transient child spec restarts only on an abnormal exit. Because this fail-fast exit's reason is :shutdown, a :transient parent will not restart it — if you chose :transient expecting automatic recovery from this condition, it will not happen, and the instance stays down until something else intervenes.

A single internal fault that gets absorbed locally (say, a transient crash in one piece of the internal machinery) does not trigger this path and does not trip a shutdown-leave drain — it is contained and retried internally, invisibly, exactly like any other supervised crash. Only exhausted local recovery surfaces as the instance's own exit.

The tables are the dashboard

There is no separate metrics endpoint: the five Postgres tables described in the adapter page are the observability surface, one SELECT away. Substitute your instance's name, stringified, for $1 in every query below.

Cluster membership and status:

SELECT node, status, joined_at FROM fief_members WHERE ns = $1 ORDER BY node;

Live leases — who the arbiter currently believes is alive:

SELECT node, expires_at, granted_at FROM fief_leases
WHERE ns = $1 AND expires_at > now() ORDER BY node;

A member present in fief_members with status active but absent from this query is a node whose lease has expired — the planner will reassign its vnodes (or already has) the moment it next runs.

Namespace epoch and term — bumps on every membership or ownership write; watch it moving to confirm the cluster is making progress at all:

SELECT epoch, max_term, partitions, impl_fingerprint FROM fief_meta WHERE ns = $1;

Vnode ownership — the full routing table, including open transfers (prev_owner IS NOT NULL):

SELECT vnode, owner, prev_owner, epoch FROM fief_table
WHERE ns = $1 ORDER BY vnode;

To see only what's currently mid-transfer (bounded by max_concurrent_transfers — see tuning):

SELECT vnode, owner, prev_owner FROM fief_table
WHERE ns = $1 AND prev_owner IS NOT NULL ORDER BY vnode;

Current leader — who is running the planner:

SELECT holder, term, expires_at FROM fief_leader WHERE ns = $1;

Forcing a leader change

Because leadership is judged by lease expiry exactly like membership, forcing a handover during an incident (a leader stuck on a bad host you want to evacuate, say) is a single UPDATE that expires the current seat:

UPDATE fief_leader SET expires_at = now() WHERE ns = $1;

This is safe by construction: the next standby to campaign wins a strictly higher term (the term ratchet), and if the old holder is actually still alive and tries to write with its old term, that write is rejected as stale. Nothing about vnode ownership or lease data is touched — only who is allowed to run the rebalance loop next.

Per-failure-mode playbook

Each row of the netsplit matrix maps to a concrete "what you'll see, what's safe, what to do":

  • A node is unreachable from its peers, but reaches Postgres. fief_members still shows it active with a live lease; it keeps serving its vnodes normally. Callers on the wrong side of the network split see timeouts, never wrong answers. Nothing to do — this heals when the network does. (Row 1.)
  • A node has lost Postgres reachability specifically (but is otherwise up). It self-fences at lease_ttl − margin on its own clock and stops serving; its lease then expires from Postgres's point of view at lease_ttl, and the planner reassigns its vnodes with no donor — expect a burst of cold-start rebuild activity on survivors right after that gap closes. Safe to leave alone; if the node is confirmed dead rather than transiently partitioned, there is nothing extra to do — the takeover already happened by the time you'd intervene. (Row 2.)
  • Postgres itself is down or unreachable cluster-wide. No joins, leaves, or rebalancing can happen; fief_meta's epoch stops moving. Under the default on_arbiter_loss: :freeze, a short outage is invisible (nodes keep serving their last-observed table); an outage longer than the fence window takes the whole cluster unavailable, but consistent — nothing reassigns while blind. What to do: restore Postgres. There is no Fief-side action that helps while the arbiter is down, and nothing about cluster state is at risk in the meantime. (Row 3.)
  • The leader is gone or cannot reach Postgres. Rebalancing and failure reassignment pause; routing, leases, and self-fencing are entirely unaffected. A standby takes over by term automatically — this is a scheduling gap, not an incident. If it is not resolving on its own within a few leadership_ttl windows, SELECT holder, term, expires_at FROM fief_leader ... to confirm no one is actually campaigning (a total outage of every candidate node, say), rather than forcing a change with no one able to win it. (Row 4.)
  • Presence (the push hint channel) is severed. No visible symptom beyond slightly slower convergence — everything still reaches the same end state on the poll cadence instead of on a push. Not an incident; nothing to do. (Row 6.)

For the full mechanics and guarantees behind each of these, read failure and recovery and the netsplit matrix — this section only names what an operator watching the tables above should expect to see and whether it warrants action.

Verified by

  • test/fief/node/shutdown_drain_test.exs — describe "graceful teardown" (leave runs against a live kernel and releases the lease before the tree falls; idempotent after an explicit leave/1; a timed-out drain is killed at the deadline and degrades to the failure path), describe "shape gating" (default timeout 20_000 ms; when the sentinel is absent), describe "restart isolation" (an internal crash is contained — no leave, no sentinel churn), and describe "auto_shutdown" (an internal supervisor's exhausted recovery brings the whole instance down with reason :shutdown — the fail-fast contract this page describes).
  • Fief.Supervisor and Fief.Node.ShutdownDrain moduledocs — authoritative for the shutdown mechanics and the fail-fast rationale.
  • test/fief/authority/postgres_test.exs — describe "migrations" and the Fief.StateStoreContract suite fix the column names and shapes the SELECTs above assume.
  • Configuration — the leave_on_shutdown option's full spec (type, default, shape gate).

Design notes: docs/leave-on-shutdown.md (the shutdown-leave decision record: rationale, the two-layer supervision tree, the fail-fast contract); docs/design.md §9 (observability: "the tables are the dashboard").