Replicant.Snapshotter.Incremental (Replicant v0.2.0)

Copy Markdown View Source

The incremental-backfill chunk reader (spec §4): a spawn_link'ed process (dies with the Connection — parent-spec closeout §17 precedent) on its OWN Postgrex connection.

Loop per chunk: AssemblerServer.open_snapshot_window/2 (tracking starts BEFORE LW — spec §2 R1; the deferred reply is also the §4 pacing gate) → capture LW → keyset SELECT → capture HW → deliver_snapshot_chunk/2 (deferred reply = backpressure). Progress tokens ride each chunk (ctx.progress); on {:error, :window_reset} (an in-process reconnect, from open's released deferred call OR a deliver) the loop RE-READS durable progress and continues — forward progress is monotone. Completion is a dedicated empty final chunk with complete?: true, re-delivered until the :complete token is durable (spec §6.3).

PK-less tables (spec §6.4): whole-table unit inside one REPEATABLE READ txn, provisional streamed batches; contention ⇒ redo with first?: true on the attempt's FIRST batch; 3 attempts ⇒ halt :snapshot_table_contended.

Resume is re-discovered, never token-trusted (Critical Rule 2)

On BOTH the initial resume and every reconnect-reload, table metadata (qualified, pk_quoted, pk_raw, col_quoted, col_type_names) is re-discovered from the server (quote_ident/format('%I.%I') via QueryBuilder.pk_columns/0 + table_columns/0). The persisted progress token is attacker-writable (a row in the progress store) and decode/1 validates only that pk_quoted/pk_raw are LISTS — NOT that their elements are safe identifiers. So the token is used ONLY for the resume POSITION (which tables are done, which is current, the PK bound, and the terminal flag), matched to freshly-discovered tables by QUALIFIED NAME (a value compared, never interpolated). No token-sourced identifier ever reaches keyset_chunk SQL, so a tampered token can never inject SQL — the M2 injection defense is fully closed. A tampered BOUND is a DIFFERENT axis and is NOT convergence-safe: a valid-arity but numerically-HIGHER bound skips the rows below it (a data-SKIP, not a re-read). But setting that bound needs progress-store WRITE access — the same trust tier as corrupting the sink directly — so it is out of scope for the injection defense.

Value-free boundary (Critical Rule 1)

Every Postgrex/decode fault — raised OR returned — scrubs to %Replicant.Error{reason: :snapshot_failed} (shape-only), exactly the v1 Snapshotter's discipline. PK bounds are row values: they live ONLY in the token and the bound parameters, never in telemetry, errors, or logs.

Summary

Functions

Spawn + LINK the reader to the caller (the Connection). Returns the pid.

Types

args()

@type args() :: %{
  slot_name: String.t(),
  connection: keyword(),
  publication: [String.t()],
  sink: module(),
  mode: :sink_owned | :lib,
  snapshot: keyword(),
  resume: Replicant.SnapshotProgress.t() | nil,
  floor_lsn: Replicant.lsn()
}

table_ref()

@type table_ref() :: %{
  schema: String.t(),
  table: String.t(),
  qualified: String.t(),
  pk_raw: [String.t()],
  pk_quoted: [String.t()],
  col_quoted: [String.t()],
  col_type_names: %{optional(String.t()) => String.t() | non_neg_integer()}
}

Functions

start(args)

@spec start(args()) :: pid()

Spawn + LINK the reader to the caller (the Connection). Returns the pid.