One field's pass: batches of rows, probed, rewritten, checkpointed.
This is ADR-0002 decisions 4, 5 and 6 in one place. The engine
(Encryptor.Ecto.Migrator) decides what to visit; a pass is how one
{schema, field, prefix} is visited, which is also exactly the key its
checkpoint row is written under.
The order of operations for one row, and why it is that order
- The source column is
NULL- nothing to do, and no key is touched. - Probe the target (decision 5): attempt the
totype's load on the bytes already in the target column. If it succeeds the row is in the target state and is skipped. Probe-first is what makes the whole pass idempotent by construction, which in turn is what makes the checkpoint a performance record rather than a correctness one. - Load through the source (
from:, ADR-0004 decision 2), and, where the field declared one, applyvalidate:to what it loaded. A failure of either is:undecryptable: the row cannot be read in a way anything trusts, and an operator has to decide what that means. - Dump through the target, under the row's own tenant.
- Compare and swap (decision 4): the update is conditional on the target column still holding the exact bytes step 2 read. Zero rows affected means the application wrote the row while the migrator was working on it - not an error, and not something to retry into a lost update. The row is re-probed and counted as concurrently migrated.
A dry run does every one of those except the swap, which is what makes it an exact rehearsal including the decrypt and the encrypt cost (decision 7).
The third mode reads and stops
mode: :verify (decision 10) does steps 1 to 3 and stops there. It does not
dump, because the encrypt would produce bytes nothing writes, and the
classification does not need them: decision 7 defines :migratable as the
probe failing and the from load succeeding, which step 3 has already
settled. It never opens a transaction and never records a checkpoint, for
the same reason a dry run does not.
A verification also visits differently. sample: n reads one random n
rows per field instead of paging the table (Keyset.sample_query/6, which
records why the sample is random rather than the first n in key order),
and records no cursor: a random draw has no "how far it got" to report.
What an unauthenticated source changes here
A field that declared source_authenticated: false (ADR-0004 decision 3)
has its migratable rows counted :migratable_unverified instead - the same
work, a different word in the evidence, because no authentication tag ever
confirmed those bytes. The class is a property of the field rather than of
the row, so it is decided once per pass and applied wherever a row would
otherwise be counted :migratable, the concurrent-write arm included.
validate: is the host's own check on the loaded plaintext, run before the
value is re-encrypted (decision 3b) and in every mode, because a
verification that skipped it would call a row migratable that a write would
refuse. It runs against the loaded value and never sees the report: a
rejection is :undecryptable with the reason :validate_rejected, and a
raise from it is {:raised, Module} like any other, so neither arm can put
a plaintext anywhere.
The batch is the transaction, and a halt discards it
Each batch is one transaction and the checkpoint row is written inside it,
so the cursor and the rows it describes are consistent by construction.
Under on_error: :halt - the default - a failing row rolls the batch back
rather than committing the rows before it: committing them without a
checkpoint is harmless, but committing them with one would advance the
cursor past the failing row, and the next resume would skip the very row
that stopped the pass. Probe-first makes redoing the discarded work free.
Nothing here holds a value longer than a row
The plaintext of one row exists between step 3 and step 4 and is never
logged, inspected, put in an exception, or carried into the report (ADR-0002
decision 11). The failures the report keeps carry the primary key, the
schema, the field, and a reason already reduced to atoms and module names by
Encryptor.Ecto.Migrator.Source.
Summary
Functions
Which checkpoint row this pass owns (ADR-0002 proposed amendment 6).
The cursor this field resumes from, or nil for a full scan.
Runs one field to the end of its table, or to the row that halts it.
Types
@type t() :: %Encryptor.Ecto.Migrator.Pass{ batch_size: pos_integer(), checkpoint: :table | :none, checkpoint_table: String.t(), except_tenants: [String.t()], field: atom(), from_source: Encryptor.Ecto.Migrator.Source.resolved(), key: Encryptor.Ecto.Migrator.Keyset.key(), mode: Encryptor.Ecto.Migrator.pass_mode(), on_error: :halt | :continue, only_tenants: [String.t()] | nil, plan: module(), prefix: String.t() | nil, progress: (Encryptor.Ecto.Migrator.Report.t() -> any()), repo: module(), sample: pos_integer() | :all, schema: module(), source: String.t(), source_authenticated: boolean(), source_column: atom(), target_column: atom(), tenant: Encryptor.Ecto.Migrator.Plan.tenant(), tenant_column: atom() | nil, to: module(), to_arity: 1 | 3, to_params: term(), validate: (term() -> term()) | nil }
Everything one field's pass needs, resolved once before it starts.
:validate is typed by what it may return rather than by what it is
contracted to return. The contract is
Encryptor.Ecto.Migration.field_spec/0's (term() -> boolean()); this is
a function the host wrote, arriving through a compiled plan, and a pass that
declared the contract here would be asserting a fact about someone else's
code that nothing checked. validate/2 checks it instead.
Functions
@spec checkpoint_key(t()) :: Encryptor.Ecto.Migrator.Checkpoint.key()
Which checkpoint row this pass owns (ADR-0002 proposed amendment 6).
The cursor this field resumes from, or nil for a full scan.
resume: false returns nil without reading anything, which - because of
probe-first - is always a legal thing to do.
@spec run(t(), Encryptor.Ecto.Migrator.Report.t(), term()) :: {Encryptor.Ecto.Migrator.Report.t(), :ok | :halt}
Runs one field to the end of its table, or to the row that halts it.
Returns the report and :ok, or the report and :halt where a failure
stopped the pass under on_error: :halt.