The primary key the pass pages over, or the reason it cannot.
iex> alias Encryptor.Ecto.Migrator.Keyset
iex> Keyset.primary_key(Encryptor.Ecto.TestSchemas.Card)
{:ok, {:id, :integer}}Keyset pagination over one schema's table, below the schema layer.
ADR-0002 decision 6: rows are visited in primary-key order with
where: r.id > ^cursor, order_by: r.id, limit: ^batch_size, never
OFFSET, which degrades quadratically and skips rows when the set shifts
under it.
Why the queries are schemaless
ADR-0002 decision 3 puts the migrator below the schema layer: it reads the
raw bytes of the ciphertext columns and calls the type modules itself,
with params it constructs. A query built on the schema module would run
every encrypted field through its own load/3 on the way out - which is to
say it would decrypt the column the migrator is here to re-encrypt, under
whatever tenant happened to be in scope, and raise before the pass ever
reached its own probe. So the queries here name the table's source
string, and what comes back is what the adapter returned.
Which primary keys are ordered, and which are refused
ADR-0002 proposed amendment 4, answering Q6 with the operator's ruling
integer/binary PKs day one, composite documented-unsupported until asked
for: a single-column primary key of an integer or a binary type is ordered
day one, because both are a total order every supported adapter expresses as
one > comparison on one column, and the binary case covers UUID keys
stored as :binary_id.
A composite primary key, or a single column of a type with no such order, is
refused with a clear error naming the schema and its primary key rather
than paged over under a guess. There is deliberately no order_by: escape
hatch: reopening this is additive when a real host arrives with a
composite-key table.
The cursor is the primary key as the adapter returned it, and it is compared with the same type it was read at - which is why the type travels beside it everywhere in this module rather than being re-derived from the value's shape.
Summary
Types
The primary key of a schema the migrator can page over: its column and the Ecto type its comparisons are made at.
Functions
One batch of rows, as [primary_key, source_value, target_value] lists.
The primary key the pass pages over, or the reason it cannot.
Re-reads one row's target column, for the re-probe after a lost swap.
One random sample of rows, in the same shape batch_query/7 returns.
The table the rewrite reads and writes, prefix included where one was given.
The compare-and-swap update for one row (ADR-0002 decision 4).
Restricts a query to the tenants a run named (ADR-0002 decision 11).
Types
@type key() :: {atom(), :integer | :binary_id | :binary}
The primary key of a schema the migrator can page over: its column and the Ecto type its comparisons are made at.
Functions
@spec batch_query( String.t(), key(), atom(), atom(), atom() | nil, term(), pos_integer() ) :: Ecto.Query.t()
One batch of rows, as [primary_key, source_value, target_value] lists.
tenant_column is the column a tenant_from rewrite reads the tenant off,
and nil for a rewrite whose tenant is :none or a resolver module; where
it is given, the tenant is appended to each row.
source_column and target_column are the same column for an ordinary
rewrite and different ones for the backfill leg of an adoption migration
(into:), which is why both are selected even when they coincide.
@spec row_query(String.t(), key(), term(), atom()) :: Ecto.Query.t()
Re-reads one row's target column, for the re-probe after a lost swap.
@spec sample_query(String.t(), key(), atom(), atom(), atom() | nil, pos_integer()) :: Ecto.Query.t()
One random sample of rows, in the same shape batch_query/7 returns.
ADR-0002 decision 10's sample: option. The rows are drawn in random
order rather than as the first size rows in key order, and the reason is
the whole value of the option: key order is the order the pass writes in, so
a prefix of it is precisely the region a partial or resumed pass migrated
first. A sampled verification over that prefix would report every row
:already_target while the tail of the table was untouched - a verification
that is wrong in the one direction a verification must not be wrong in.
The cost is an ordering over the scope, which is why sample: is a drift
detector a host runs on a schedule and sample: :all - the keyset scan
batch_query/7 performs - is the acceptance test at the end of a rotation
(ADR-0004 decision 8, step 6).
random() is spelled that way by PostgreSQL and SQLite; MySQL spells it
rand(). That is the same adapter assumption ADR-0002 decision 10's census
SQL already makes, and it is confined to this one clause.
The table the rewrite reads and writes, prefix included where one was given.
The compare-and-swap update for one row (ADR-0002 decision 4).
The update is conditional on the target column still holding the exact bytes the migrator read. Zero rows affected means the application wrote the row while the migrator was working on it, which is not an error and is not a clobber: the row is re-probed and counted.
@spec tenant_filter(Ecto.Query.t(), atom(), [String.t()] | nil, [String.t()]) :: Ecto.Query.t()
Restricts a query to the tenants a run named (ADR-0002 decision 11).
The filter is a where on the tenant column rather than a decision made per
row: a crypto-shredded tenant's rows are permanently undecryptable by
design, and the point of the filter is that the pass never visits them and
still exits zero.