Guarantees

Copy Markdown View Source

For anyone deciding whether to trust Fief with a workload. Assumes only the overview.

Fief makes four guarantees. None of them is stated here without its scope condition, because an ownership guarantee quoted without its scope is how distributed systems disappoint people. Each guarantee ends with a Verified by block naming the deterministic tests that exercise its edges — a claim with a linked repro is a different class of claim — and a Design notes pointer into the repository's internal documentation for the derivation.

1. Single ownership

At most one node owns a given key at any instant — including during netsplits, node crashes, leader failures, and mid-rebalance. This guarantee has two layers with different scope, and the difference matters:

  • Routing single-ownership is unconditional. No message is ever processed by a node that is not the key's current owner under a live lease. This holds regardless of fencing mode and regardless of how the vnode implementation behaves, because it is enforced by the framework on the receiving side: every inbound message is validated against the receiver's own authoritative ownership state before delivery, and a non-owner answers {:moved, ...} instead of executing anything. The scope is exactly the routed path: for Fief.Key, handle_message/3 runs only under validated ownership, while a key's own handle_info/2 ingress (timers, subscriptions) is explicitly outside the validation, bounded instead by the fence kill — see the two mailboxes.
  • State and side-effect single-ownership is per surface. For Fief.Key it holds under on_fence: :terminate, the default: at most one live process for a key exists anywhere in the cluster, so its timers and side effects fire on at most one node. The weaker fencing modes — :continue and :callback — deliberately trade this layer for availability, and each states exactly what it forfeits on the fencing-modes page. For Fief.Cache this layer is unconditional: entries live only in a table owned by the vnode's serving process and die with it — there is no fencing policy to weaken, and a cache entry has no timers or side effects to fire.

Two consequences worth internalizing. First, a buggy or malicious key module cannot break routing single-ownership for anyone — the validation is not its job. Second, if you configure on_fence: :continue, the sentence you are entitled to quote is only the routing half.

Verified by:

  • test/fief/vnode_test.exs — describe "ownership validation": non-owners answer moved, never deliver.
  • test/fief/netsplit_test.exs — describe "row 1: node↔node partition, arbiter reachable": one owner per vnode throughout a partition.
  • test/fief/keyed_netsplit_test.exs — describe "row 1: node↔node partition, arbiter reachable — escheat gating (⊨ rule 1)": a recipient that cannot reach the donor keeps the key unavailable rather than starting a second copy.
  • test/fief/key/vnode_impl_test.exs — describe "planned transfer (freeze / pull / grant / inject / settle)": exactly one live process per key across a handover (the old process retires inside the handover, before the new one starts).
  • test/fief/peer_test.exs — the disconnect_node scenario: a real dist partition with both sides holding database access produces failed calls on the severed side, never split-brain service.
  • test/fief/cache_test.exs — the drop-on-handoff describe: cache entries do not survive an ownership move — the new owner starts cold rather than ever serving alongside the old one.

Design notes: docs/design.md §2, §6.3; docs/implementation.md §6.2 (the two-layer statement). The transfer protocol behind the mid-rebalance case is model-checked (specs/FiefTransfer.tla); see verification.

2. Agreed ownership

All nodes converge on the same view of ownership, derived from a single authoritative table held in the arbiter (Postgres, in the default deployment). There is no gossip round, no CRDT merge, and no conflict resolution step, because there is nothing to resolve: ownership is derived — owner(key) = table[hash(key)] — and the table has exactly one writer discipline (compare-and-swap on a monotonic epoch).

A node's local copy of the table is explicitly a hint, and may be stale without harm: senders route on it, receivers validate against truth, and a stale sender is corrected ({:moved, ...} with a table delta) rather than obeyed. Convergence is therefore a performance property, not a safety property — every hint channel in the system can be severed and the answers stay correct, just slower.

Verified by:

  • test/fief/router_test.exs — describe "call": the moved/retry loop patches stale caches and lands on the true owner.
  • test/fief/netsplit_test.exs — describe "row 6: presence severed": hints lost, correctness unaffected, reaction degrades to poll tempo.
  • test/fief/peer_test.exs — the join scenario: two real nodes converge on one table through the shared authority.

Design notes: docs/design.md §4 ("receivers hold truth; senders hold hints"), §5.4.

3. Downtime over inconsistency

A node that cannot prove its ownership is current stops serving — before anyone else is allowed to take over. Every member holds a TTL lease it must renew against the arbiter. A node whose renewal fails, or whose arbiter is unreachable, self-fences at TTL − margin on its own monotonic clock, unilaterally, with no communication required; peers may treat the lease as dead only at TTL as judged by the arbiter's clock. The asymmetric margin is sized for renewal round-trip plus twice the clock-drift bound — the 1× margin has a model-checked counterexample.

The consequence you plan operations around: keys become unavailable; they never become doubly owned. The lease TTL is the floor on failover time — recovery from a node death takes at least TTL plus one planner reaction, and no hint channel can beat it, because a crash and a partition are indistinguishable from the outside. The same coin's other side: a key can be unavailable by design — mid-transfer with the donor unreachable, or with the arbiter down — and this is Fief choosing its stated preference, not a bug. The netsplit matrix enumerates every partition case and what you observe in each.

What "stops serving" does to your key processes is per-key policy: the default kills them (a muted-but-alive process still fires timers and side effects, so muting is not fencing). See Fencing modes.

Verified by:

  • test/fief/node_test.exs — describe "self-fencing" and describe "margin arithmetic": the fence fires at TTL − margin from renewal send time, before any peer may act.
  • test/fief/netsplit_test.exs — describe "row 2: node↔arbiter partition" and describe "row 3: arbiter down (on_arbiter_loss)".
  • test/fief/keyed_netsplit_test.exs — describe "row 2: node↔arbiter partition": the fenced node's keys stop; survivors take over only after arbiter-judged expiry.
  • test/fief/peer_test.exs — the kill -9 scenario: takeover on the real database clock, never before lease expiry.

Design notes: docs/design.md §4 (leases with self-fencing), §6.4; the margin arithmetic is model-checked in specs/FiefLease.tla.

4. Scale in keys, not nodes

Millions of keys are supported; the node count is deliberately modest. Per-key state is never replicated or agreed upon — the only shared state is the vnode→node assignment table, a few kilobytes regardless of key count, and key→PID mappings (Fief.Key) or cache entries (Fief.Cache) live in plain local state on the owning node (a PID is the worst possible thing to replicate; Fief never does). The arbiter is consulted for leases, membership, and rebalancing — never on the message hot path.

The scope condition: Fief assumes a full distribution mesh and targets roughly ≤ 50–60 nodes. Beyond that, distributed Erlang itself becomes the constraint. If you need hundreds of nodes, Fief is the wrong shape; if you need millions of keys on tens of nodes, this is the intended sweet spot.

Verified by: this one is architecture rather than a runtime edge — the claim rule's honest label is by construction: nothing in the codebase replicates per-key state (the wire envelope carries opaque payloads; the authority schema in Fief.Authority.Postgres.Migrations has no key-shaped table to put it in).

Design notes: docs/design.md §2, §4, §9 (cluster scale).

What Fief never promises

Stated as loudly as the guarantees, because these are the questions the guarantees invite:

  • No delivery guarantees. A node crash vaporizes mailboxes, including messages Fief already routed correctly. Fief.Key.call/5 — or Fief.Cache.put/4 — returning {:error, :timeout} means unknown outcome, not "not executed." Callers that need at-least-once processing implement acks and retries above Fief — and since they must do so anyway, the moved/retry transfer protocol costs them almost nothing extra. Every Fief message is epoch-tagged, so building idempotent, retry-safe callers on top is natural. (One deliberate wrinkle: a key may bundle in-flight callers into its handover blob at freeze and answer them from its next incarnation via Fief.Key.reply/2 — but the callers' timeouts keep ticking meanwhile, so a late reply is a bonus, never a promise.)
  • No cross-owner message ordering during transfers. While a key's ownership moves, one sender's messages may be processed partly by the old owner and partly by the new one without a per-sender ordering guarantee across the pair. Exactly-one-owner-at-a-time still holds throughout.
  • No HA arbiter (v1). Postgres is an accepted single point of failure: arbiter down means no membership or ownership changes, and — under the default on_arbiter_loss: :freeze — nodes fence within a lease TTL. The cluster becomes unavailable; it does not become inconsistent. The netsplit matrix has the full row.

Verified by: test/fief/router_test.exs — describe "call" (the {:error, :timeout | :exhausted} return shapes); test/fief/netsplit_test.exs — describe "row 3: arbiter down (on_arbiter_loss)".

Design notes: docs/design.md §3, §7.