Instances

Copy Markdown View Source

For users setting up an instance who want to understand the namespace/coordination-domain model. Assumes getting started and ownership.

Fief has no global singleton. Nothing runs under the :fief application itself; you start instances in your own supervision tree, Ecto/Oban-style, and every piece of configuration is an instance option — there is no application environment anywhere. An instance is the unit of coordination: a namespace, a set of nodes, a vnode table, a lease discipline.

# in MyApp.Application.start/2 children:
{Fief,
 name: MyApp.Fief,
 authority: {Fief.Authority.Postgres, repo: MyApp.Repo},
 vnode_impl: {Fief.Key.VnodeImpl, []},
 partitions: 1024,
 lease_ttl: 5_000}

The name is the coordination domain

name is not a label — it is the namespace, the identity of the coordination domain. It must be identical on every node participating in the instance, and it scopes every row Fief writes to the arbiter: leases, membership, the vnode table, leadership, and the immutable-config record. Two nodes started with the same name against the same database are the same instance and coordinate ownership with each other; two nodes with different names share nothing, even in one database.

Because the namespace scopes every row, many instances coexist on one database — adding an instance is rows, never new DDL (one set of Fief tables serves them all; see operating the Postgres adapter). And because everything an instance owns derives from its name — every process, every ETS table — many instances coexist in one BEAM without colliding.

Design notes: docs/implementation.md §1 (the instance-model decision: the name is the namespace, scopes every Authority row), §2 (every process and ETS name derives from the instance name).

Instances live in your supervision tree

An instance is a child spec you place in your tree and supervise like any other. You start it, you stop it, you decide where it sits relative to your repo and the rest of your app.

Under that one child spec, the instance is two layers, and the split is why the failure story below holds together:

  • A thin outer supervisor is what your child spec actually starts. It holds two children: the kernel below, and — only when configured — a graceful-shutdown sentinel beside it (the leave_on_shutdown option — see configuration for the setting and operations for what it buys you at deploy time).
  • The kernel is a rest_for_one supervisor holding everything that actually serves: impl-owned infrastructure and the routing cache first (so they survive a restart below them), then the lease/fencing state machine, then the planner and the vnode agents. When the lease/fencing process crashes, its lease and ownership state can no longer be trusted, so rest_for_one tears down and rebuilds everything after it through the join path — a crashed node fences itself as a side effect of supervision, with no special code. What sits ahead of it (the routing cache, impl-owned state) rides out that restart untouched.

A Fief node going down means your child spec exits. The inner layer recovers from routine internal faults on its own restart budget; it does not retry forever. If it exhausts that budget, it exits, and that takes the whole instance down with it — the outer process exits too, deliberately, rather than attempting a supervision strategy that has already just failed. Fief does not paper over a sick instance with a silent internal reboot. Surfacing as your child spec exiting means your supervision tree makes the call: a :permanent child restarts and rejoins fresh through the normal join path; :transient or :temporary lets it stay down. This is deliberate, not a gap — the verified recovery path for a dead node is the cluster-level one (lease expiry, another node taking over), not an invisible reboot underneath you.

Two practical consequences follow from instances being yours to place:

  • Mixed vnode implementations run side by side. A Fief.Key instance and a Fief.Cache instance — a differently-shaped workload, a coherent cache in place of process lifecycles — are two instances, not one instance with modes. Each is its own coordination domain with its own table.
  • Timing and policy are per instance. lease_ttl, the fencing margin, the arbiter-loss posture, transfer pacing — all are per-instance options, so a Fief.Key instance and a Fief.Cache instance can run side by side with different failover floors in the same BEAM against the same Postgres.

Design notes: docs/implementation.md §1 (instance-based, not a global singleton; mixed impls and per-instance tuning), §2 (the two-layer tree and why rest_for_one inside it is load-bearing); docs/leave-on-shutdown.md §2 (the outer/inner split and the fail-fast contract).

Partition count is immutable — repartition blue/green

The vnode count (partitions) is fixed for the life of the instance. The first node ever to start the namespace writes it — alongside the vnode impl's fingerprint — to the instance's config record; every later joiner is checked against it, and a mismatch is fatal at startup, before the node touches shared state. This is deliberate: keys hash to vnodes with a pinned function (see ownership), so changing the vnode count would remap every key at once — the double-ownership hazard, cluster-wide.

So repartitioning is not a config edit. It is a blue/green operation, and the instance model is what makes it routine rather than a rewrite:

  1. Stand up a second instance (a new name) with the new partition count, alongside the running one.
  2. Migrate keys at the application level — dual-write, or drain-and-cut per key — from the old instance to the new one.
  3. Retire the old instance once it is empty.

For a Fief.Cache instance, step 2 degenerates to nothing: drop-on-handoff means the new instance simply repopulates from misses, with no application-level migration to write.

The one irreversible decision of the ownership layer becomes a green-field deployment beside a live one. The same story covers changing the hasher — the shared Fief.Hasher substrate, immutable for the identical reason and applying identically to a Fief.Cache instance — see Keys are binaries.

Verified by:

  • test/fief/node_test.exs — describe "join": the first node's config is written first-writer-wins; a joiner with mismatched partitions is fatal before shared state, naming the field.
  • test/fief/key/vnode_impl_test.exs — describe "join validation": a joiner whose key-layer fingerprint disagrees with the namespace's recorded config is rejected at join and never enters membership.
  • test/fief/key_test.exs — describe "config_fingerprint/1 ({:hasher, module})": the fingerprint is {:hasher, module}, a fun hasher or a hasher without hash_key/1 is fatal, so the recorded value is a stable comparable term.
  • test/fief/cache_test.exs — describe "Fief.Cache.VnodeImpl.config_fingerprint/1 ({:hasher, module})" and describe "join fingerprint (cache-flavored; mechanism proved generically elsewhere)": the same fingerprint and join-mismatch behavior, exercised from the second surface.

Design notes: docs/implementation.md §9 (partitions and the impl fingerprint written to the config record by the first node, every joiner compares and refuses to start on mismatch; the blue/green repartitioning story), §1; docs/design.md §9 (partition count is the one irreversible decision).

What this means for you

  • You own the lifecycle. An instance is a child spec in your tree; you start, stop, and place it. There is no global Fief to configure and no application environment to thread through releases.
  • Run as many as the workload wants. One BEAM and one database can host several instances — different vnode impls, different failover floors — each a self-contained coordination domain that shares nothing with the others but the physical tables.
  • Pick partitions as if it were permanent, because it is. Sizing it up front is the cheap path; the escape hatch — stand up a new instance and migrate — exists and is clean, but it is a data migration, not a setting you flip. Read tuning before choosing anything other than the default.