Changelog

Copy Markdown View Source

All notable changes to capstan are documented here. The format follows Keep a Changelog, and this project adheres to Semantic Versioning.

Unreleased

0.2.0 - 2026-07-22

Added — C2 initial snapshot

A consistent backfill of the rows that pre-exist a pipeline's start, woven into the continuously-running C1 stream so there is no gap and no duplicate at the snapshot→stream handoff — proven live under concurrent load, resumable mid-backfill, strict-once in normal operation, fail-closed on every silent-loss condition, Rule 1 end-to-end. Additive: absent the :snapshot config, a pipeline is pure C1, byte-for-byte.

  • Public API — a snapshot: [tables:, store:, chunk_size:] block on Capstan.start_link/1; a new optional Capstan.Sink.handle_snapshot/2 (required only in snapshot mode) delivering a concrete list of %Capstan.Change{op: :snapshot} + a value-free Capstan.Snapshot.Meta; upsert-by-PK sink semantics (a hard precondition). A new Capstan.SnapshotStore behaviour (read/1+write/2 over a %Capstan.Snapshot.State{}, InMemory reference impl) persists the per-table PK cursor, separate from the GTID checkpoint.
  • Mechanism (ADR-0005) — cursor-gated suppression + a brief per-chunk LOCK TABLES … READ that captures an exact GTID position G (a provable lower bound on the chunk's read view). The read-only lock-free @@gtid_executed bracket was rejected: it leads InnoDB row-visibility under concurrent commit (a probe-confirmed silent corruption). The stream advances the GTID watermark; the snapshot advances the PK cursor — two authorities, never conflated (the snapshot adds no GTID; ADR-0001 untouched). No C1b, no ADR-0004 supersession — the P0 pre-seed makes the dump and the watermark agree by construction.
  • New source privilege — the query account needs LOCK TABLES on the snapshot tables (in addition to SELECT); no RELOAD/FTWRL. Added to scripts/capstan-preflight.sql.
  • Fail-closed snapshot halts (distinct value-free reason each, tripwired): order-faithful-PK only (:snapshot_pk_unsupported_type; a collation-ordered string PK is refused — ROADMAP C2a), :snapshot_table_no_primary_key, :snapshot_table_not_captured, :snapshot_lock_unavailable, :snapshot_schema_drifted, :snapshot_source_mismatch (@@server_uuid across both connections), :snapshot_chunk_read_failed, :snapshot_bootstrap_gtid_read_failed, :snapshot_coordinator_down, :snapshot_config_drifted (the configured snapshot.tables no longer match a durable :complete/mid-snapshot state — a table added or removed after a state exists halts rather than silently never backfilling it), the SnapshotStore faults, and {:snapshot_sink_error, _}. A tables: :all snapshot is refused :config_invalid — an explicit table list is required (ROADMAP C2b).
  • Delivery guarantee — strict-once in normal operation; the only duplicate window is a crash between a chunk's {:ok} and the durable pk_cursor persist (the one in-flight chunk re-emits, C1's bounded posture; an upsert-by-PK sink converges). A second durable high-water, delivered_pk, is persisted BEFORE each chunk emit and backstops a DELETE that lands in that same window: on restart the cursor-gate forwards a streamed delete of an already-delivered key (k ≤ delivered_pk) so the row is swept, never left as a phantom. Effect-once across the crash window is the deferred sink-owned path (ROADMAP C1a).

Docs & tooling

  • A test-environment guide (docs/testing.md) and a runnable minimal example consumer (examples/print_consumer.exs). ADR-0005 now ships in the generated documentation.
  • Removed a reference to a private repository from the README and ADR-0001 (the downstream-consumer example is now stated generically).

0.1.0 - 2026-07-21

Added — C1 streaming spine

The initial streaming spine: connect to MySQL as a replica, tail the row-based binary log from a GTID position, assemble committed transactions, deliver them to a sink, and durably advance a processed-GTID checkpoint — halting fail-closed on every condition that could otherwise lose or corrupt data silently.

  • Public API — Capstan.start_link/1 and child_spec/1; a supervised pipeline (Capstan.Connection owning the socket, Capstan.AssemblerServer owning assembly, delivery, and the checkpoint). Bad config is refused before any socket opens.
  • Position and dedup model (ADR-0001) — GTID set is the sole authoritative and sole persisted position; dedup by set membership via the public Capstan.Gtid (member?/2 and the interval algebra). No scalar ordinal; %Capstan.Position{} file/pos are diagnostic only.
  • Fail-closed server preconditions (ADR-0002) — a connect-time gate on binlog_format, binlog_row_image, binlog_row_metadata, binlog_row_value_options (PARTIAL_JSON refused), and gtid_mode, each with a distinct value-free reason; table_id-keyed schema resolution; ROWS_QUERY_LOG_EVENT decoded and discarded (Rule 1); TLS on by default with an explicit peer-verification choice required (:tls_verification_unspecified otherwise).
  • Transaction shape and checkpoint semantics (ADR-0003) — three terminators (XID, QUERY("COMMIT"), self-committing DDL QUERY); XA refused honestly as :unsupported_transaction_shape with the buffer discarded and the checkpoint held; the checkpoint as a processed-GTID watermark that advances on delivered and filtered transactions; server_id-conflict discrimination on error 1236 (:server_id_conflict, not a livelock).
  • Sink contract — Capstan.Sink with handle_transaction/1, handle_schema_change/2, and checkpoint/0; changes typed Enumerable.t() (single-pass safe); DDL statement text redacted to structured schema/table/kind.
  • Lib-owned checkpointing — Capstan.CheckpointStore behaviour + InMemory reference implementation, persisting one processed gtid_set string per pipeline.
  • Rule 1 (value-free) — row values, DDL literals, ROWS_QUERY SQL, and the connection password never reach a log line or telemetry payload; proven red-capable on all four vectors.
  • Telemetry — connection, transaction, schema-change, and gap events with a value-free metadata allowlist.
  • Streaming liveness — a replication heartbeat requested from the server (:heartbeat_period_ms, default 15 000) plus a parent-side liveness window (:stream_timeout_ms, default 60 000; must exceed the heartbeat period or start-up fails closed), so a silent half-open partition can no longer hang the pipeline: the timeout emits [:capstan, :connection, :stream_timeout], kills the blocked reader, and reconnects; a persistent partition halts as :stream_stalled. TCP keepalive is the OS-level backstop.

Scope (ADR-0004)

C1 ships lib-owned checkpoint mode and resume-from-durable-checkpoint (start_position: :checkpoint) only. Sink-owned checkpoint mode and explicit start positions (%Capstan.Position{} override, :current) are deferred and refused fail-closed (:sink_owned_mode_unsupported, :start_position_override_unsupported, :start_position_current_unsupported).