Architecture

For contributors — and anyone who wants to know how the pieces fit before reading code. Assumes the concepts pages. This page is also the reading guide into the internal documents in docs/, which stay in the repository rather than on hexdocs.

The layering in one paragraph

The framework kernel provides consistent vnode ownership and nothing else: leases and self-fencing, the epoch-fenced vnode table, the planner, one agent process per owned vnode, a session-validated peer channel between successive owners of a vnode, and fencing as a supervision kill. The kernel is key-agnostic — it routes opaque messages to vnodes and never interprets their contents. Everything key-shaped — hashing, the per-key process model, incremental migration — lives in Fief.Key, the sanctioned implementation of the Fief.Vnode behaviour; Fief.Cache is a second, deliberately trivial implementation of the same behaviour. The guarantee statement follows the cut: routing single-ownership is the kernel's property and holds unconditionally for any impl; state and side-effect single-ownership is the impl's property, and Fief.Key under on_fence: :terminate is the impl that delivers it (the guarantees).

The component stack

Two structural decisions worth knowing before reading code. Instances, not a singleton: nothing runs under the :fief application; users start instances in their own tree, every process and ETS name derives from the instance name, and all configuration is instance opts (instances). The model-checked rules are implemented once: every transfer rule with a counterexample behind it lives in Fief.Transfer's two pure machines — structs and event functions returning {effects, machine}, no processes, no clock reads — so the rules are property-testable without a cluster and embeddable without re-derivation.

The supervision tree

MyApp.Supervisor                        (the user's tree)
└── {Fief, name: MyApp.Fief, ...}       → Fief.Supervisor, strategy: one_for_one
    │                                     auto_shutdown: :any_significant
    ├── Fief.Kernel                     significant: true, restart: :temporary
    │   │                                 strategy: rest_for_one
    │   ├── Fief.Key.Limbo              (impl-requested root children — fence survivors)
    │   ├── {authority adapter}         e.g. Fief.Authority.Postgres
    │   ├── Fief.Router.Cache           ETS owner, boring
    │   ├── Fief.Node                   lease + fencing + join state machine
    │   ├── Fief.Planner.Supervisor     leadership: elector + planner slot
    │   └── Fief.Vnode.Manager          DynamicSupervisor
    │       └── Fief.Vnode.Agent        one per owned vnode; root of that vnode's world
    │           └── (impl-owned, linked)  e.g. Fief.Key's DynamicSupervisor of key servers
    └── Fief.Node.ShutdownDrain         shutdown-leave sentinel; last, shape-gated (below)

Fief.Supervisor — the process the user's tree actually holds a child spec for — is a thin outer supervisor over exactly two children: Fief.Kernel, which holds all of an instance's live machinery, and Fief.Node.ShutdownDrain, the shutdown-leave sentinel appended last. The outer's own strategy is :one_for_one with auto_shutdown: :any_significant; the kernel is mounted significant: true, restart: :temporary, so the outer never restarts it — a kernel exit is handled by fail-fast, not retry (below).

rest_for_one inside the kernel is load-bearing. If Fief.Node crashes, its lease and fencing state cannot be trusted, so everything ordered after it within the kernel — every agent, every impl subtree — is torn down and restarted through the join path. A crashed Fief.Node fences the node as a side effect of supervision, with no special code.

Fencing is a kill, not a request. Fief.Node fences by having the manager brutally kill the agents; links take the impl subtrees with them. The impl's fence callback is advisory and runs under a deadline — the kill happens regardless, which is what makes on_fence: :terminate unconditional: user code cannot be alive past the fence by being slow. The corollary discipline: impl resources must live under the agent's link tree (processes; ETS tables die with their owning process) precisely so the kill is complete.

Fail-fast: a kernel exit takes the whole instance down. The kernel is itself a rest_for_one supervisor with its own restart budget, so by the time the kernel process exits it has already exhausted local recovery — an outer retry would only replay a strategy that just failed. Instead, the kernel being significant plus the outer's auto_shutdown: :any_significant means a kernel exit brings the whole instance down: Fief.Supervisor itself exits, with reason :shutdown. A Fief node going down therefore surfaces as its top process exiting — not a silent internal reboot — and it is the user's supervision tree that decides what happens next, via whichever restart type (:permanent / :transient / :temporary) it gives the {Fief, opts} child spec. This is by design: the verified recovery path for a sick node is the failure protocol (lease expiry, takeover by a healthy peer), not a respawn racing the same fault, and a persistently sick instance should become visibly dead quickly rather than flap in place.

The sentinel sits outside the kernel's restart unit, on purpose. Were Fief.Node.ShutdownDrain a flat tail child of the kernel's rest_for_one instead, an ordinary internal fault — a Fief.Vnode.Manager blip — would restart it and run a full graceful-leave drain while a perfectly healthy Fief.Node is still alive underneath it. Sitting beside the kernel as an outer, one_for_one-supervised sibling, it is only ever torn down for a genuine instance teardown (System.stop/0, SIGTERM, the user's supervisor terminating {Fief, opts}) or after a kernel exit — and in the latter case the node it would drain is already gone, so its leave/1 no-ops and returns at once. The sentinel is present only under a shape gate — leave_on_shutdown enabled (default: on, 20 s), a :vnode_impl configured, the planner subtree enabled, and the instance not sim-joined — since without all four there is either nothing to drain or nothing that could ever drain it. See configuration for the option; docs/leave-on-shutdown.md carries the full rationale.

No per-vnode supervisor layer. The agent is the root of its vnode's world directly; an impl supervisor started from the agent monitors its parent, so agent death tears down the whole vnode subtree by construction. An agent crash is recovered by the manager restarting it fresh — local keys cold-start on next touch, the same recovery path as a single key process crashing, since ownership never changed.

Limbo sits first, deliberately. Key processes under the weaker fencing modes (:continue / :callback) must survive the fence kill, so they are supervised outside the agent link trees, at the kernel's root, ordered first under the kernel's rest_for_one — survivors outlive a restart of any later kernel child and die only when the kernel itself does (instance teardown, or the fail-fast exit above). The limbo pair (its supervisor and its tracker) is one_for_all: if either crashes, both restart into an empty, mutually consistent state, so survivors die with their bookkeeper rather than outliving the only process that could still reap them — a crash-equivalent loss inside :continue's documented best-effort posture, instead of a double-serve hole. Fencing modes carries the user-facing consequences.

The determinism seams

Everything nondeterministic goes through one compile-bound module, Fief.Seam: every clock read and every deadline (lease basis, fence timer, planner cadence, pull retries, sweep ticks), every jitter draw, every cross-node send and registered-name minting, plus checkpoint/2 — a macro marking sim-visible progress points that compiles to a literal :ok in prod builds. Seams are compile switches rather than instance options because no user ever configures them and they sit on hot paths; prod builds have zero indirection and cannot ship a sim module by accident.

The payoff is a FoundationDB-style deterministic simulation harness: under test, seam timers, seam transport deliveries, and Fief.Authority.Local operations all feed one event queue owned by one scheduler, and virtual time advances only at quiescence. Arbiter writes become schedule points interleavable between any send and its delivery — "the planner's CAS lands between the grant being sent and received" is a line in a test, not a hope pinned on sleeps — and those interleavings are exactly where the model-checked rules earned their counterexamples. The discipline that keeps the harness honest is the no-raw-timers rule: any bare Process.send_after, gen_statem state timeout, or receive-after punches a hole in virtual time, so mix seams greps lib/ for the whole family of leaks and CI fails on any hit. Crude and sufficient.

The scope limit, stated plainly: this is determinism at the protocol level. Intra-node BEAM scheduling, monitors, and supervisor restarts stay real and concurrent, so a green simulation run is not full determinism — which is why a small :peer suite (test/fief/peer_test.exs) still drives real distributed nodes against a real database: real dist semantics and real clocks, for exactly what simulation deliberately cannot capture. The sim seam implementations are real-unless-hooked — they delegate to the real implementations unless a scheduler is registered — so the same test-compiled beams behave identically to prod on those nodes.

The rest of the testing shape in one sentence each: contract suites are the executable specs for the authority and vnode behaviours, run against every adapter and impl; protocol tests run several instances in one BEAM against a shared Fief.Authority.Local under the scripted scheduler, which is where the netsplit matrix's deterministic repros live; and TLA+ stays authoritative for the protocol — code tests verify the implementation follows the spec, they never re-derive the rules (verification).

Reading guide: the internal documents

The repository's docs/ directory is the contributor tier, in decision-record register — linked from these guides, never required to use the library:

• docs/design.md — the normative what and why: guarantees, protocols, the model-checked rules and their counterexample history. When a guide says "design notes," it points here.
• docs/implementation.md — the where and how in code: layout, supervision, behaviour contracts, wire shapes. Its lockstep rule is the contract: when a module there and a module in lib/ disagree, one of them is a bug.
• docs/key-redesign.md — the decision record for the current Fief.Key callback contract (the single init/3 entrypoint, the open mailbox, the two-phase freeze).
• docs/build-order.md — the lab notebook: milestone-by-milestone as-built notes. Mined for caveats, never published.
• specs/ — the TLA+ models and the action↔code map (specs/README.md); verification is the public tour.

Design notes: docs/implementation.md §0–§2 (layering, layout, supervision), §10 (seams), §11 (testing shape).