One field's pass: batches of rows, probed, rewritten, checkpointed.

This is ADR-0002 decisions 4, 5 and 6 in one place. The engine (Encryptor.Ecto.Migrator) decides what to visit; a pass is how one {schema, field, prefix} is visited, which is also exactly the key its checkpoint row is written under.

The order of operations for one row, and why it is that order

1. The source column is NULL - nothing to do, and no key is touched.
2. Probe the target (decision 5): decide whether the bytes already in the target column are in the target state, and skip the row if they are. Probe-first is what makes the whole pass idempotent by construction, which in turn is what makes the checkpoint a performance record rather than a correctness one. See "Two ways to probe" below for which of the two answers the question.
3. Load through the source (from:, ADR-0004 decision 2), and, where the field declared one, apply validate: to what it loaded. A failure of either is :undecryptable: the row cannot be read in a way anything trusts, and an operator has to decide what that means.
4. Dump through the target, under the row's own tenant.
5. Compare and swap (decision 4): the update is conditional on the target column still holding the exact bytes step 2 read. Zero rows affected means the application wrote the row while the migrator was working on it - not an error, and not something to retry into a lost update. The row is re-probed and counted as concurrently migrated.

A dry run does every one of those except the swap, which is what makes it an exact rehearsal including the decrypt and the encrypt cost (decision 7).

Two ways to probe, and when the cheap one is allowed

Decision 5 wrote the probe as a load attempt: call the to type's load on the stored bytes, and read success as "already in the target state". That is one decrypt per already-migrated row, and on a table that is mostly migrated

• a resumed pass, a second run, a scheduled re-run - it is the whole cost of the pass. The same decision says the probe short-circuits to a header inspection wherever upstream can classify a message without a key, which assumption A9 resolved at acceptance: Encryptor.Message.describe/1 reads a message's encryption context keylessly.

So a pass whose target is one of this package's own vault-backed types probes by reading the header, in two steps.

Step one is a comparison against what the target declares. The context the message claims must equal the context that target's declaration writes - the vault's static pairs, the declared "table" and "column", and whatever :context added, which Encryptor.Ecto.Binary.declared_context/1 composes once rather than twice - and the algorithm suite the message names must equal the one the target's vault is configured to write. The "tenant_ref" pair the vault derives is compared for presence and not for value: which tenant a row belongs to is not what the probe asks.

Comparing the whole context rather than merely parsing the header is what keeps the context-change rewrite correct - from: and to: naming the same module with different params (Encryptor.Ecto.Migration's "Silence is allowed only where authentication is provable"). Both sides of that rewrite write well-formed messages of this package's format, and a probe that read no further than "it parses" would call every unrewritten row already migrated and silently do nothing.

Step two is a proof, because a context and a suite do not identify a key. ADR-0002's R3 rewrites a column whose "format, algorithm, library, or encryption context" changes, and two of those - a different vault over the same declared context, a re-keyed one - leave every compared pair identical while the bytes are still the source's. What separates them is the wrapping key, which the header names as each encrypted data key's {provider_id, key_name} and which nothing keyless can predict: the name is a keyed derivation the provider mints (Encryptor.Key.Aes), so the pass cannot compute the one the target would use for a row it has not written.

It can prove one instead. The first row of a batch claiming a given {suite, encrypted data keys} identity is loaded rather than believed, and only an identity a load has just proven the target reads is allowed to short-circuit the rest of that batch. A source row's identity is never proven - its load fails, exactly as it does on main - so an R3 rewrite whose two sides differ only in vault, key or suite rewrites every row it used to rewrite. The saving is per batch rather than per row: one decrypt for each distinct wrapping key a batch touches, instead of one per already-migrated row.

The proof is sound because a key name is bound to its material forever - Encryptor.Key.Aes makes reusing one for different material a defect, since it silently breaks every message already written under it. Two messages with the same identity and the same context are therefore readable by the same key, and the first one's load answers for both.

The memo lives in the batch's own fold and nowhere else. It is a pure optimization: dropping it costs decrypts, never correctness, which is why it is scoped to the smallest thing that still pays - a batch is one transaction, and a pass that resumes has no use for what a previous transaction proved.

describe/1's answer is an unverified claim by whoever wrote the bytes, and that is the right strength here: nothing downstream of the probe is an authorization decision (Encryptor.Message's own warning). The worst a forged header can do is have the pass leave a row alone. That is not what the load probe would have done with the same row - a load that fails sends the row to the source reader, which rewrites it from the intact source - so the header probe trades a rewrite the load probe would have performed for the decrypts it saves, and the row waits until a mode: :verify run, which always loads, reports it.

Two cases keep the load attempt:

• a foreign target - a plain Ecto.Type, someone else's parameterized type, or a vault that is not running when the pass is built - because there is no header this package can read a claim out of;
• a verification (mode: :verify), because opening the bytes is the whole of what decision 10 makes it the authoritative answer for. A verification that classified from headers would be a cheap census wearing the acceptance test's name, and the cheap census already exists (Encryptor.Ecto.Migrator.Census).

Both ways are the one probe/2 below, which is what Encryptor.Ecto.Migrator.verify/2 means by borrowing the pass's probe rather than reimplementing it.

The third mode reads and stops

mode: :verify (decision 10) does steps 1 to 3 and stops there. It does not dump, because the encrypt would produce bytes nothing writes, and the classification does not need them: decision 7 defines :migratable as the probe failing and the from load succeeding, which step 3 has already settled. It never opens a transaction and never records a checkpoint, for the same reason a dry run does not.

A verification also visits differently. sample: n reads one random n rows per field instead of paging the table (Keyset.sample_query/6, which records why the sample is random rather than the first n in key order), and records no cursor: a random draw has no "how far it got" to report.

What an unauthenticated source changes here

A field that declared source_authenticated: false (ADR-0004 decision 3) has its migratable rows counted :migratable_unverified instead - the same work, a different word in the evidence, because no authentication tag ever confirmed those bytes. The class is a property of the field rather than of the row, so it is decided once per pass and applied wherever a row would otherwise be counted :migratable, the concurrent-write arm included.

validate: is the host's own check on the loaded plaintext, run before the value is re-encrypted (decision 3b) and in every mode, because a verification that skipped it would call a row migratable that a write would refuse. It runs against the loaded value and never sees the report: a rejection is :undecryptable with the reason :validate_rejected, and a raise from it is {:raised, Module} like any other, so neither arm can put a plaintext anywhere.

The batch is the transaction, and a halt discards it

Each batch is one transaction and the checkpoint row is written inside it, so the cursor and the rows it describes are consistent by construction. Under on_error: :halt - the default - a failing row rolls the batch back rather than committing the rows before it: committing them without a checkpoint is harmless, but committing them with one would advance the cursor past the failing row, and the next resume would skip the very row that stopped the pass. Probe-first makes redoing the discarded work free.

Nothing here holds a value longer than a row

The plaintext of one row exists between step 3 and step 4 and is never logged, inspected, put in an exception, or carried into the report (ADR-0002 decision 11). The failures the report keeps carry the primary key, the schema, the field, and a reason already reduced to atoms and module names by Encryptor.Ecto.Migrator.Source.