ADR-0005: Spill is ephemeral non-fsync'd scratch, not a durable WAL

Status: Accepted Date: 2026-08-12 Deciders: replicant maintainer (1.0 hardening D8; records the replicant-streaming-spill decision that shipped in 0.1.0)

Context

A single in-progress streamed transaction can be larger than memory. Without a spill path the bounded in-flight window halts fail-closed at the first oversized transaction — a hard ceiling on transaction size. Spilling the transaction to disk introduces an at-rest surface: spilled frames are :erlang.term_to_binary({subxid, %Change{}}), i.e. cleartext row values for the duration of the transaction. Two design questions follow:

1. Is spill a durable WAL (fsync'd, crash-surviving) or ephemeral scratch?
2. Is the spill file encrypted at rest?

Decision

• Spill is ephemeral non-fsync'd scratch. Frames are written to a per-txn file under a 0700 spill directory with 0600 files, length-prefixed, write-once-read-once within a single transaction's lifetime, deleted on commit/abort/reset/halt, and swept per-slot on every reconnect. There is no fsync — spill is not a WAL; a crash re-streams the transaction from the durable confirmed_flush LSN (the spill is discarded, never replayed). (lib/replicant/spill.ex, lib/replicant/spill/reader.ex.)
• Two ceilings, not one: resident RAM is bounded by max_inflight_lag (the spill trigger), and disk is bounded by max_spill_bytes (a transaction exceeding it halts :spill_exhausted). The §4 in-flight numerator is received − floor − spilled compared to max_inflight_lag + max_spill_bytes, so a legitimately-spilling transaction is not double-counted toward the RAM halt.
• No in-lib encryption. Key management is a separable subsystem with its own operational surface (KMS, rotation, envelope keys) that this library does not own. The operator places the spill dir on a secure/encrypted volume and cleans it on decommission; the default dir is a 0700 subdir of the OS temp dir. (lib/replicant/spill.ex, documented in the README.)
• Value-free on fault. Any spill File.* / binary_to_term fault scrubs to a fixed Replicant.Spill.Error{reason: :spill_io_failed} (no bytes). binary_to_term uses [:safe] (no atom/function creation) and shape-validates the frame; the snapshot-resume-token path additionally rejects the compressed external-term format before deserialization (a decompression-bomb guard). (lib/replicant/spill/reader.ex, lib/replicant/snapshot_progress.ex.)

Options Considered

Option A: fsync'd durable spill (treat it like a WAL)

Cons: Spill is NOT a WAL — its correctness comes from re-streaming on crash, not from surviving one. fsync'ing would impose disk-latency on the hot path for no correctness gain (the durable anchor is the slot's confirmed_flush, not the spill file). Rejected.

Option B: In-lib blanket encryption of spill files

Cons: Key management is its own operational subsystem (provisioning, rotation, envelope encryption). Embedding it here would couple a CDC library to a KMS and force a key-handling contract on every consumer. The 0600/0700 permissions + scratch-lifetime + operator-placed encrypted volume is the documented mitigation. Rejected as a library responsibility; noted as a clean future addition (a persistent dir is the operator's to secure). See the 1.0 P2 backlog.

Option C: Ephemeral non-fsync'd scratch, no in-lib encryption (adopted)

Trade-off Analysis

The adopted design treats spill as scratch whose only job is to keep a too-large transaction out of RAM until its StreamCommit. Correctness does NOT depend on the spill surviving a crash — it depends on the durable confirmed_flush watermark + idempotent sink, which already exist. Paying fsync or encryption cost would optimize a non-load-bearing property. The residual cleartext-at-rest risk during a spill is real and documented; it is the operator's to mitigate via the spill dir placement (the default OS-temp location is cleared by the OS).

Consequences

• Easier: Oversized transactions deliver effect-once without halting; no fsync latency on the hot path; no key-management surface in the library.
• Harder: Spilled row values are cleartext at rest for the duration of one transaction. This is a deployment-time responsibility (point dir at a secure volume) and is documented in the README "Operator guidance" and noted in the 1.0 P2 backlog (in-lib blanket encryption is a clean future addition).
• Revisit if: the threat model includes an attacker who can read the owner's filesystem mid-spill AND the operator cannot place dir on an encrypted volume — then in-lib encryption (Option B) becomes the right call and is additive (a streaming: [spill: [encrypt: true]] knob over a key-management subsystem).