Where a pass records how far it got, and what it refuses to do about the table it records into.
ADR-0002 decision 6: after each batch the migrator records
{plan, schema, field, last_id, counts, started_at, updated_at} in a
checkpoint table, and run/2 with resume: true starts after the recorded
cursor. Decision 5 is what makes that merely a performance record: the pass
probes every row before rewriting it, so losing a checkpoint costs a
re-scan and never correctness.
The key carries the prefix
ADR-0002 proposed amendment 6 (status: proposed, ece-l6t) adds the prefix
to the key, and this module is built to that text. Without it, a caller
looping run/2 over prefixes - which is exactly what the record tells a
host with several to do - has every prefix sharing one checkpoint row, so
the second prefix resumes at the first's cursor and silently skips every
row below it. Probe-first idempotence does not cover that: it makes a
re-visited row safe, and this is a row never visited.
A prefix of nil - the repo's own default prefix - is stored as the empty
string rather than as NULL, because a NULL component defeats the unique
index in every adapter that treats NULLs as distinct, which is the same
silent-skip failure by a different route.
The table is the host's, and this package will not create it
ADR-0002 decision 9: no CREATE TABLE, at runtime or from a task. The table
arrives as generated migration source the host reads, commits, and runs with
its own mix ecto.migrate (mix encryptor.ecto.gen.migration, ece-5qb).
So preflight/2 checks and refuses, naming the generator; it is the one
place where creating the table would be convenient, which is exactly why the
line is drawn here.
checkpoint: :none keeps the alternative available as a documented degraded
mode: no checkpoint at all, every run a full scan, which decision 5 makes
correct rather than merely tolerable.
Its shape
One row per field per prefix, with a unique index over
{plan, schema, field, prefix}:
| Column | |
|---|---|
plan, schema, field | the plan module, the schema module, and the field, as text |
prefix | the schema prefix visited; "" for the repo's default |
last_id | the primary key of the last row of the last committed batch, rendered as text |
counts | the classification counts so far, as a map |
started_at, updated_at | when the pass began and when this row was last written |
last_id is text because ADR-0002 proposed amendment 4 admits both integer
and binary primary keys and one checkpoint table serves both. Rendering and
parsing are this module's (render_cursor/2, parse_cursor/2), and the
key's type travels with it so the value is parsed back at exactly the type
it was compared at.
A dry run never writes one
record/5 is called only in :write mode. A dry run that recorded a cursor
would let a later write-mode run resume past rows it never wrote - a silent
skip produced by the rehearsal, which is the one thing a rehearsal must not
do. Reading a checkpoint in a dry run is fine and is what makes the
rehearsal resemble the run.
Summary
Functions
The table name a host gets unless it names another.
The cursor a previous pass over this field and prefix recorded, if any.
Parses a stored last_id back into a cursor, or nil when it cannot.
The empty string a default prefix is stored as.
Confirms the checkpoint table exists, and refuses when it does not.
Records one batch's cursor and counts, in the transaction that wrote it.
Renders a primary key as the text the checkpoint row stores.
Types
Functions
@spec default_table() :: String.t()
The table name a host gets unless it names another.
iex> Encryptor.Ecto.Migrator.Checkpoint.default_table()
"encryptor_ecto_migration_checkpoints"@spec fetch_cursor(module(), String.t(), key(), Encryptor.Ecto.Migrator.Keyset.key()) :: term() | nil
The cursor a previous pass over this field and prefix recorded, if any.
A row whose last_id cannot be parsed back at the key's type is treated as
absent: the pass re-scans, which decision 5 makes correct, rather than
paging from a cursor nobody can vouch for.
@spec parse_cursor(String.t(), Encryptor.Ecto.Migrator.Keyset.key()) :: term() | nil
Parses a stored last_id back into a cursor, or nil when it cannot.
iex> alias Encryptor.Ecto.Migrator.Checkpoint
iex> Checkpoint.parse_cursor("42", {:id, :integer})
42
iex> Checkpoint.parse_cursor("not a number", {:id, :integer})
nilThe empty string a default prefix is stored as.
iex> Encryptor.Ecto.Migrator.Checkpoint.prefix_text(nil)
""
iex> Encryptor.Ecto.Migrator.Checkpoint.prefix_text("tenant_a")
"tenant_a"Confirms the checkpoint table exists, and refuses when it does not.
Returns :ok, or {:error, message} naming the generator. The refusal is a
message rather than a CREATE TABLE for the reason ADR-0002 decision 9
gives; the caller raises it, once, before the first batch of the first
field.
Records one batch's cursor and counts, in the transaction that wrote it.
Written by the same transaction as the batch it describes, so the two are consistent by construction - a cursor written beside a transaction rather than inside it disagrees with the rows whenever a crash lands between them.
@spec render_cursor(term(), Encryptor.Ecto.Migrator.Keyset.key()) :: String.t()
Renders a primary key as the text the checkpoint row stores.
iex> alias Encryptor.Ecto.Migrator.Checkpoint
iex> Checkpoint.render_cursor(42, {:id, :integer})
"42"
iex> Checkpoint.render_cursor(<<1, 2>>, {:id, :binary})
"0102"