Capstan.Snapshot.PrimaryKey (Capstan v1.2.1)

Copy Markdown View Source

The order-faithful primary-key core of the initial snapshot (C2, design Ch3/Ch4).

A chunk pages by ORDER BY pk in MySQL while the cursor-gate classifies streamed changes by k ≤ cursor in Elixir. For that pairing to be correct — no silently mis-classified change (delivered vs suppressed → gap/dup) — the PK's Elixir term-order MUST provably match MySQL's ORDER BY. This module enforces that with a positive type allowlist and supplies the two operations the coordinator needs on PK values:

  • canonical/2 — the equality form for the sink ledger. Canonicalization happens in Elixir (reconciliation-equality), NOT MySQL collation. The text-protocol string form of a PK (a chunk read) and the binlog-decoded integer/binary form (a streamed change) canonicalize to the SAME term, so the ledger reconciles a chunk row against a streamed change to the same key.
  • compare/2 — the ordering for the cursor-gate (k ≤ cursor). Over the allowlist it is exactly MySQL ORDER BY: integers order numerically, BINARY/VARBINARY order byte-wise (shorter prefix first), and composites order lexicographically as tuples (matching MySQL row-value ORDER BY).

The positive allowlist (Ch3.2 / Ch4; the string arm is ADR-0012)

ACCEPT the types whose comparison against MySQL ORDER BY is provably faithful:

  • signed/unsigned integer — TINYINT/SMALLINT/MEDIUMINT/INT/BIGINT
  • BINARY / VARBINARY (byte-ordered)
  • CHAR / VARCHAR — collation-ordered, with the server as the only collation oracle: a string PK's canonical form is its collation WEIGHT BYTES (probe-proven: weight-byte order == ORDER BY for ai_ci, both _bin weight forms, PAD SPACE over distinct keys, multi-level as_cs, and composites). The chunk read selects WEIGHT_STRING(pk) alongside the row; the stream side resolves weights through resolve_weights/4 (a COLLATE-pinned CONVERT(X'..' USING charset) introducer over the binlog's raw column bytes — an unpinned introducer computes the charset-DEFAULT collation's weights, a different order space). Distinct PK values are collation-distinct by the PK constraint itself, so weights are a faithful key identity.
  • composites of the above (tuple compare; a string position compares by its weight)

REFUSE everything else with :snapshot_pk_unsupported_type:

  • the TEXT family — order semantics are consistent, but a TEXT prefix PK pays a measured filesort per chunk page (Using filesort vs VARCHAR's Using index; probe/collation_weight_probe.exs Q6b) — superlinear backfill.
  • ENUM/SET — the column's ORDER BY is member-position order and its column weights are position-based, while any stream-side introducer computes STRING weights — two disagreeing order spaces, no uniform mechanism (probe Q13).
  • DECIMAL/DOUBLE/FLOAT/DATE/DATETIME/temporal and any other type — their Elixir term-order diverges from MySQL.

BIGINT UNSIGNED across 2^63 (Ch3.1)

An unsigned 64-bit PK spans [0, 2^64). A signed-wrap decoder reads the high half as negative (2^63 → -2^63, 2^64-1 → -1), so it would order 2^63… before 0… and the cursor-gate would mis-classify a high-half key. canonical/2 decodes to the TRUE unsigned value: a text form parses directly, and a raw integer that arrived signed-wrapped (a negative for an unsigned column) is normalized back into [0, 2^width).

No PK → fail closed

introspect/2 reads information_schema for the PRIMARY KEY. A table with no PK accepts a UNIQUE key fallback ONLY IF every column of that key is NOT NULL — a nullable unique-key column admits NULL-keyed rows that WHERE k > cursor never selects (a silent gap, Ch3.3), so such a key is refused :snapshot_table_no_primary_key. A table with neither is refused the same way.

Rule 1

No PK value is ever logged or telemetered here. Errors are value-free atoms. The information_schema queries embed only the schema/table (structural identity — the same fields telemetry may carry), never a row value, and are escaped as SQL string literals.

Summary

Types

A canonical PK: a single value (single-column PK) or a tuple (composite PK).

The introspected key: the ordered PK columns, their order-faithful types, and — for the string columns — the charset/collation pair that pins every weight and cursor literal (ADR-0012). A non-string column's charset/collation entry is nil (mirroring information_schema.COLUMNS, where CHARACTER_SET_NAME/COLLATION_NAME are NULL for non-string columns).

An order-faithful PK column type atom (integer family, :binary, :varbinary, :char, :varchar).

Functions

Builds the canonical equality form of a PK from pk_types and the raw column values (raw_values, one per PK column in order).

Compares two canonical PK values (from canonical/2): :lt, :eq, or :gt.

Introspects the PRIMARY KEY (or a NOT-NULL-complete UNIQUE key fallback) of {schema, table} over a Capstan.Query handle.

The canary verdict: the collation-ordered weight sequence equals the weight-ordered one. Ties are inherently safe — collation-equal values carry equal weights, so both sequences show the same bytes at tie positions.

Builds the bootstrap order-contract canary SQL for one string pk column.

The pure key resolution over the two information_schema resultsets — the RED-capable seam introspect/2 delegates to (so the PK / unique-key fallback / type-refusal logic is provable without a live server).

Classifies a single column's (DATA_TYPE, COLUMN_TYPE) to an order-faithful pk_type(), or refuses :snapshot_pk_unsupported_type.

Resolves the collation weight bytes for a batch of raw PK-column values over a Capstan.Query handle — {:ok, %{raw => weight}}, or a value-free {:error, reason}.

Builds the COLLATE-pinned weight-resolution SQL for a batch of raw column values.

Types

canonical_pk()

@type canonical_pk() :: integer() | binary() | tuple()

A canonical PK: a single value (single-column PK) or a tuple (composite PK).

key()

@type key() :: %{
  pk_columns: [String.t()],
  pk_types: [pk_type()],
  pk_charsets: [nil | String.t()],
  pk_collations: [nil | String.t()]
}

The introspected key: the ordered PK columns, their order-faithful types, and — for the string columns — the charset/collation pair that pins every weight and cursor literal (ADR-0012). A non-string column's charset/collation entry is nil (mirroring information_schema.COLUMNS, where CHARACTER_SET_NAME/COLLATION_NAME are NULL for non-string columns).

pk_type()

@type pk_type() ::
  :tinyint
  | :tinyint_unsigned
  | :smallint
  | :smallint_unsigned
  | :mediumint
  | :mediumint_unsigned
  | :int
  | :int_unsigned
  | :bigint
  | :bigint_unsigned
  | :binary
  | :varbinary
  | :char
  | :varchar

An order-faithful PK column type atom (integer family, :binary, :varbinary, :char, :varchar).

Functions

canonical(pk_types, raw_values)

@spec canonical([pk_type()], [binary() | integer()]) :: canonical_pk()

Builds the canonical equality form of a PK from pk_types and the raw column values (raw_values, one per PK column in order).

A raw value is either a text-protocol string (a chunk read) or a binlog-decoded term (a streamed change); both forms of the SAME PK canonicalize equal. Integers normalize to their true value (an unsigned column's signed-wrapped negative is unwrapped into [0, 2^width)); binaries pass through their bytes; a string column's value is the {raw, weight} pair its caller assembled (the weight being the canonical bytes — ADR-0012; a bare binary for a string column raises, value-free, rather than silently comparing raws). A single-column PK returns a bare value; a composite returns a tuple in column order.

compare(a, b)

@spec compare(canonical_pk(), canonical_pk()) :: :lt | :eq | :gt

Compares two canonical PK values (from canonical/2): :lt, :eq, or :gt.

Over the allowlist this is exactly MySQL ORDER BY: Erlang term order coincides with MySQL numeric order for integers, byte-wise order for BINARY/VARBINARY, and — for same-arity composites — lexicographic tuple order for row-value ORDER BY. Both arguments must be canonical forms; the caller compares like against like (a table's PK shape is fixed).

introspect(query, arg)

@spec introspect(
  Capstan.Query.t(),
  {String.t(), String.t()}
) :: {:ok, key()} | {:error, atom()}

Introspects the PRIMARY KEY (or a NOT-NULL-complete UNIQUE key fallback) of {schema, table} over a Capstan.Query handle.

Returns {:ok, %{pk_columns: [...], pk_types: [...]}} for an order-faithful key, or a value-free {:error, reason}: :snapshot_pk_unsupported_type when a key column's type is outside the allowlist, :snapshot_table_no_primary_key when there is no PK and no NOT-NULL-complete unique key. A transport/query fault propagates its value-free reason from Capstan.Query.query/2.

order_contract_ok?(by_collation, by_weight)

@spec order_contract_ok?([[binary()]], [[binary()]]) :: boolean()

The canary verdict: the collation-ordered weight sequence equals the weight-ordered one. Ties are inherently safe — collation-equal values carry equal weights, so both sequences show the same bytes at tie positions.

order_contract_sql(charset, collation, order_by)

@spec order_contract_sql(String.t(), String.t(), :collation | :weight) :: String.t()

Builds the bootstrap order-contract canary SQL for one string pk column.

Two queries over the same fixed ASCII vector, introduced with the column's own charset + collation pin: the first orders by the COLLATION (ORDER BY v), the second by the WEIGHT BYTES (ORDER BY WEIGHT_STRING(v)), each returning one hex-encoded weight per row. Under MySQL's documented order contract the two weight sequences are EQUAL (order_contract_ok?/2) — the gate's every decision rests on that equality.

resolve_key(index_rows, column_rows)

@spec resolve_key([[binary() | nil]], [[binary() | nil]]) ::
  {:ok, key()} | {:error, atom()}

The pure key resolution over the two information_schema resultsets — the RED-capable seam introspect/2 delegates to (so the PK / unique-key fallback / type-refusal logic is provable without a live server).

  • index_rowsSTATISTICS rows [INDEX_NAME, SEQ_IN_INDEX, COLUMN_NAME, NON_UNIQUE]
  • column_rowsCOLUMNS rows [COLUMN_NAME, DATA_TYPE, COLUMN_TYPE, IS_NULLABLE, CHARACTER_SET_NAME, COLLATION_NAME]

(the four STATISTICS cells are NOT NULL; the charset/collation cells are NULL for non-string columns, so those two cells may be nil). Returns the same {:ok, key()} | {:error, reason} contract as introspect/2.

resolve_pk_type(data_type, column_type)

@spec resolve_pk_type(binary(), binary()) ::
  {:ok, pk_type()} | {:error, :snapshot_pk_unsupported_type}

Classifies a single column's (DATA_TYPE, COLUMN_TYPE) to an order-faithful pk_type(), or refuses :snapshot_pk_unsupported_type.

Signedness rides COLUMN_TYPE (which carries the unsigned attribute); DATA_TYPE alone cannot distinguish it. CHAR/VARCHAR accept (ADR-0012 — the weight path carries their ordering); the TEXT family (measured per-page filesort) and ENUM/SET (position-based order no introducer weight path reproduces) refuse, as does every non-integer, non-binary, non-char/varchar type (its Elixir order diverges from MySQL).

resolve_weights(query, charset, collation, raws)

@spec resolve_weights(Capstan.Query.t(), String.t(), String.t(), [binary()]) ::
  {:ok, %{required(binary()) => binary()}} | {:error, atom()}

Resolves the collation weight bytes for a batch of raw PK-column values over a Capstan.Query handle — {:ok, %{raw => weight}}, or a value-free {:error, reason}.

The stream-side arm of ADR-0012: a binlog-decoded string PK arrives as the column's raw bytes (the row event carries no charset), and its canonical form is the weight the SERVER computes for those bytes under the column's charset + collation. The batch is chunk-bounded (128 values per statement) so one bulk-load transaction cannot exceed max_allowed_packet; a query fault surfaces as the scrubbed Capstan.Query atom for the caller to budget and halt on.

weight_sql(charset, collation, raws)

@spec weight_sql(String.t(), String.t(), [binary()]) :: String.t()

Builds the COLLATE-pinned weight-resolution SQL for a batch of raw column values.

One WEIGHT_STRING(CONVERT(X'<hex>' USING charset) COLLATE collation) term per raw, in order — the result row's columns pair positionally with raws. The COLLATE pin is LOAD-BEARING: without it the CONVERT result carries the charset-DEFAULT collation, whose weights are a different order space (probed: a column's as_cs multi-level weight vs the unpinned ai_ci primary-only weight), and the unpinned form in a WHERE additionally raises ERROR 1267 on non-default collations. Hex literals carry arbitrary bytes (NULs, quotes, backslashes, empty) with no string-literal escaping surface.