Tuning

For operators sizing a Fief cluster: partition count, lease timing, and transfer pacing. Assumes the guarantees and the full option reference; this page is the "how to choose", not the option list.

Three knobs matter more than the rest, and they fall into two very different classes. Partition count is a one-time, cluster-wide, irreversible decision. Lease timing and transfer pacing are per-node tuning you can revisit on any rolling restart — get them wrong and you pay in latency or churn, never in correctness.

Partition count: the one irreversible decision

partitions (default 1024) splits the keyspace into vnodes by an immutable hash. It is immutable cluster-wide — written by the first node ever to start the namespace, checked against every later joiner, fatal to change without stopping every node — because changing it remaps every key's home. There is no live repartitioning; it is a full stop, reconfigure, restart, not a rolling change.

Pick generously up front rather than growing into it. More vnodes than nodes is what lets the planner spread load evenly and rebalance in small increments as membership changes — too few vnodes for your eventual node count caps how finely ownership can ever be divided. 1024 is comfortable for most clusters; 4096 is reasonable headroom if you expect to scale out to many tens of nodes. There is no code-enforced ceiling, but more vnodes also means more fief_table rows and more planner bookkeeping per pass, so "as many as convenient" is not the answer either — undersizing the ring is the regret operators report more often than oversizing it. See configuration for the mismatch-at-join mechanics.

Lease TTL and margin: the one conservative timing

lease_ttl (default 5_000 ms) is the failover floor:

unavailability after a node death ≈ lease_ttl + planner reaction + per-key rebuild

No hint, no faster failure signal, and no configuration elsewhere can tunnel under the TTL term — a nodedown or a health check can only make the planner reaction term smaller, because the arbiter can never treat a node as dead before its lease has actually expired on its own clock (a "dead" node might merely be partitioned and still serving on the far side). Shorter TTLs recover faster but renew more often (renew_interval, default lease_ttl ÷ 5) and tolerate less clock drift and Postgres latency before a healthy node self-fences by mistake; 3–5 seconds is comfortable against a healthy Postgres. Size it against your arbiter's actual latency, not a round number — failure has the full timing argument.

lease_margin (default :auto = renew_rtt_budget + 2 × clock_drift_bound) is the self-fence window before nominal expiry, and it must stay strictly under lease_ttl — validated and fatal at startup otherwise, since a margin at or above the TTL would fence a node before its lease could ever be live. The :auto default already bakes in the conservative case (drift can swing between extremes across one measurement interval, so the bound is doubled, model-checked against a real counterexample at 1×); only override it if you have measured your own renewal RTT and clock drift and want a tighter number.

None of this conservatism carries over to leadership. leadership_ttl gates only the planner's fenced writes — a wrong or absent leader is safe, merely idle — so it can run as hot as its mechanism allows without the lease layer's clock-skew caution. The only place a fast leadership_ttl and campaign_interval pay for themselves is the compound failure (a member and the leader dying together), where a quick re-election trims how long that member's keys wait for someone to reassign them.

Transfer pacing: what actually bounds drain time

Two different knobs bound two different things, and conflating them is the easiest way to under-provision a deploy:

• max_concurrent_transfers (default 8) bounds open transfers — rows with prev_owner set — not CAS throughput. It exists so a large rebalance (a join, a scale-up) does not flood every node with concurrent pull traffic at once; a joiner "pulls only the over-capacity vnodes it is owed, capped by this budget" regardless of how unbalanced the cluster is.
• sweep_rate / sweep_interval (defaults 100 keys per 1_000 ms, inside the vnode_impl options) is the actual drain-time knob. Once a transfer is open, its keys move at this rate in the background (hot keys pull on first touch regardless, but the bulk of a vnode's cold keys ride the sweep) — this is what bounds how long a leave, a leader-driven rebalance, or a shutdown drain actually takes wall-clock time to finish, not the transfer count cap.

A node holding many keys per vnode needs either a higher sweep_rate or a longer window to drain within. That window is leave_on_shutdown (default 20_000 ms): the sentinel that runs a graceful leave on deploy waits for exactly this drain to finish, and a timeout is not unsafe — it degrades to an ordinary node failure and the un-handed-over keys just rebuild cold — but it does turn a should-be-clean deploy into cold-start churn. If your typical node's share of the keyspace cannot realistically drain inside the default 20 s at your configured sweep_rate/sweep_interval, raise sweep_rate, or raise leave_on_shutdown (kept comfortably under your orchestrator's termination-grace-period — see the runbook for the deploy-observable behavior and configuration for the option itself). leave_drain_interval (default 100 ms) is just the re-check cadence during that wait and rarely needs tuning on its own.

Two smaller knobs round this out: pull_retry_interval (default 1_000 ms) paces how often a stalled first-touch pull retries, and extract_deadline (default 5_000 ms, per key module — see the key lifecycle) bounds how long a single stuck key is allowed to hold up its vnode's freeze before it is killed and rebuilt fresh on the new owner, rather than stalling the whole handover indefinitely.

Rebalance cadence

rebalance_interval (default 1_000 ms, ± rebalance_jitter, default interval ÷ 10) is how often the planner recomputes the desired distribution and issues moves. It matters far less than the two knobs above: presence hints already trigger an immediate pass on most membership or ownership changes, so the interval mainly bounds how quickly a missed hint (a severed presence channel — netsplit row 6) is caught on poll alone. Lowering it trades a small amount of steady-state planner/arbiter chatter for slightly faster convergence in that fallback case; the default is a reasonable middle ground for most clusters.

Cluster scale

Fief assumes a full distributed-Erlang mesh and is sized for a modest node count carrying a very large key count — tens of nodes, not hundreds. Beyond roughly 60–100 nodes, default distributed Erlang itself (not anything Fief-specific) becomes the binding constraint; clusters at that scale should already be evaluating partial-mesh or custom-transport strategies independent of Fief.

Verified by

• test/guides/configuration_test.exs — every default named on this page (partitions, lease_ttl, renew_interval, max_concurrent_transfers, rebalance_interval/jitter, sweep_rate/sweep_interval, leave_drain_interval) asserted against the actual validator code.
• test/fief/node_test.exs — describe "margin arithmetic" (the :auto formula, and both lease_margin ≥ lease_ttl and handle_fence_deadline ≥ margin refusing to start).
• test/fief/planner_test.exs — "a joiner pulls only the over-capacity vnodes, capped by the transfer budget" and "a :leaving member drains: all its vnodes move away, budget-paced" — max_concurrent_transfers in action on the pure planning function.
• test/fief/key/vnode_impl_test.exs — "background sweep migrates an untouched key" and "the extract deadline kills a stuck key; it converges via {:not_here} → escheat" — sweep_rate/sweep_interval and extract_deadline exercised end to end.
• test/fief/node/shutdown_drain_test.exs — describe "shape gating" (leave_on_shutdown default 20_000 ms) and describe "graceful teardown" (the timeout path this section's drain-time arithmetic feeds into).

Design notes: docs/design.md §9 (operational notes: partition count, TTL tuning, transfer pacing, cluster scale).