Delivery and errors

For application developers handling {:error, ...} returns from call and cast and building retry-safe callers. Assumes getting started.

Fief guarantees routing correctness — no message is ever processed by a node that isn't the current owner — and deliberately guarantees nothing about delivery: a node crash takes mailboxes with it, and a fenced or killed owner produces no reply at all. Callers that need at-least-once processing implement acks and retries above Fief, and would have to no matter what a clustering layer promised. This page is the caller's side of that bargain: what each return value means, when "unavailable" is the system working as designed, and how to make retries safe.

What call returns

MyKey.call(instance, key, msg) returns {:ok, reply} or {:error, reason}. The reasons, and — the part that matters for retries — what each one tells you about whether the message was processed:

This table applies verbatim to Fief.Cache.get/put/delete, which forward through the same router call and pass {:error, reason} through unchanged. One cache-specific wrinkle sits outside it: Fief.Cache.get/3 also returns a bare :error for a plain cache miss, which is a normal result, not a failure — see Using Fief.Cache.

The timeout covers the entire call loop — every routing hop, the anti-stampede jitter, the authoritative fallback — so one deadline bounds the whole thing, not each attempt.

:timeout is the only unknown-outcome error. Silent drops are the contract by design: an owner that fenced, crashed, or was killed mid-message sends no error reply — there is nobody left to send it. So a timeout means any of: never delivered, delivered and lost with the node, or processed, with the reply lost. Treat every other error as "safe to retry blindly" and :timeout as "retry only if the message is idempotent."

What cast returns

:ok from cast means the message was handed to the transport — never that it was delivered, and never that it was processed. Fire-and-forget is the contract; if you need to know, use call, and if you need certainty, an ack from your own handle_message reply is the only real one. Fief.Cache has no cast: get, put, and delete are all synchronous calls, so this section doesn't apply to it.

Unavailable by design

Some error stretches are not incidents — they are downtime over inconsistency doing its job. Expect timeouts, :no_owner, or :exhausted for a bounded period when:

• A node died. Its keys are unavailable for up to the lease TTL plus a planner reaction, then recover elsewhere from durable truth (a fresh init/3) — the TTL is the failover floor.
• A key is mid-transfer across a partition. If the new owner can't pull the key's state from the old one, the key waits — a pull timeout never triggers rebuild-from-truth, because the old owner may still be alive on the far side (what callers see mid-transfer).
• The arbiter is down. Under the default on_arbiter_loss: :freeze, nodes stop serving within a lease TTL and stay down until Postgres returns (the netsplit matrix, row 3).
• A key's pending queue is full. Messages arriving during a pull are pended, bounded by max_pending_per_key (default 128); overflow surfaces as caller timeouts rather than unbounded memory.

In all four, the alternative to the error is double execution somewhere. If your dashboards treat these as anomalies, they will page you for the system's central design decision.

Building retry-safe callers

(This section is advice — patterns, not promises.)

The foundation you can lean on: routing single-ownership is unconditional, so a retry is never processed concurrently with the original by two owners. What retries can produce is sequential duplication — original processed, reply lost, retry processed by the same or the next owner. So:

• Make messages idempotent. Carry a request id and dedupe in key state, or phrase operations as absolute writes ({:set_status, :shipped}) rather than relative ones (:increment). Then the :timeout ambiguity costs you nothing: retry until you get a reply.
• Don't encode cross-transfer ordering assumptions. Per-sender message ordering across an ownership change is explicitly not guaranteed — two of your messages straddling a rebalance may be processed by different owners out of send order. If order matters, carry versions in the messages and reject stale ones in handle_message.
• Bound retries and surface the failure. Unavailable-by-design windows are real; an infinite retry loop turns a bounded unavailability into an unbounded queue. Retry with backoff for about the failover floor you've configured (lease TTL + slack), then fail up to your caller.
• Durability is your job — for Fief.Key. {:ok, reply} means your callback returned — whether the state change survives the node is decided by what you persisted, not by Fief (three ways out). Write-through inside handle_message is the simple answer; a timer-driven write-behind flush via handle_info/2 is the economical one. For Fief.Cache durability is definitionally absent: entries never survive an ownership move, by design rather than by omission — see drop-on-handoff.

Verified by

• test/fief/router_test.exs — describe "call" (the reply path, :moved patch-and-retry, bounded hops → authoritative fallback → :exhausted, whole-loop timeout with silent-drop semantics, :noconnect refresh-then-retry-once) and describe "cast and owner_of" (handed-to-transport semantics).
• test/fief/key/vnode_impl_test.exs — describe "pend bounds" (overflow → caller timeouts) and describe "escheat gating" (pull timeout never rebuilds).
• test/fief/keyed_netsplit_test.exs — describe "row 1: node↔node partition, arbiter reachable — escheat gating (⊨ rule 1)" and describe "node crash" (what callers observe through the failure and recovery windows).

Design notes: docs/design.md §7 (delivery semantics), §5.4 (the send/retry protocol), §3 (the no-delivery-guarantees non-goal); docs/implementation.md §5 (the call loop and timeout discipline).