Encryptor.Ecto.Migrator.Census (Encryptor.Ecto v0.2.0)

The SQL half of verification: what a plan's tables look like, read with no application and no key.

ADR-0002 decision 10. Encryptor.Ecto.Migrator.verify/2 above these is the authoritative answer - it opens the bytes and is therefore the acceptance test. These are the cheap ones, and they exist for two people the verifier does not serve:

the operator watching a six-hour pass, who should be able to see where it has got to without running the application against production credentials to find out;
the DBA reviewing the change, who should be able to confirm the outcome without being handed a key.

So nothing here connects to a repository, decrypts anything, or needs the vault to be running. queries/2 renders SQL text against a plan's own table and column names, and what an operator does with it is paste it into psql.

Target-or-not, and the prefix that has to be wider than one byte

The census asks whether a row is in this package's format, and answers nothing else. Distinguishing "not one of ours" from "one of theirs" would need a decoder for a format this package has no correctness obligation to (ADR-0004 decision 11), so it is not attempted and the queries are written as target-or-not.

There is a trap in the cheap version of that question, and it is the reason header_bytes/0 is four rather than one: cloak_ecto's envelope opens with a reserved 0x01 byte, and this package's messages open with a version byte of their own. A census keyed on the first byte alone can therefore read as identical across both formats - a report that says "one format, all migrated" over a table that is half legacy. Grouping on a wider prefix is what separates them.

The queries group rather than test, which is the same defensiveness one level up: the census does not assert "this prefix is ours", it prints every prefix present with a count beside it, and the operator reads which one is growing. A host on a legacy format whose fifth byte happens to collide gets a census that looks wrong rather than one that lies.

The three queries

Kind	Answers
`:format`	Which formats are in this column, and how many rows of each
`:progress`	For one tenant, how far the rotation has got
`:integrity`	Nothing became `NULL` and nothing became empty. Run before, run after, compare

:progress is emitted only for a rewrite that resolves its tenant from a column (tenant_from). A rewrite whose tenant is :none or a resolver module has no tenant column to filter on, and a per-tenant progress query over it would either be a whole-table count wearing a tenant's name or a guess about where the tenant lives.

Placeholders, and why the header is one

Two of the queries carry :placeholder names for the operator to substitute, because neither value is knowable from the plan: :tenant is whichever tenant is being watched, and :current_header is the byte prefix the target format is currently writing.

The operator gets :current_header from the :format query on the same column rather than from any key: once the new type modules are live, the prefix whose count is growing is the target one. That keeps the whole set keyless, which is the property the whole module exists for.

Dialect

PostgreSQL, which is what ADR-0002 decision 10 wrote and what this package tests against. substring(x from 1 for n), count(*) FILTER (WHERE ...) and octet_length/1 are the three constructs that are not portable to every adapter; a host on another one translates three lines, and the shape of the question is unchanged.

Summary

Types

kind()

Which question a census query answers. See the moduledoc's table.

query()

One rendered query, and enough about it to print a heading over it.

Functions

header_bytes()

How many leading bytes the format census groups on.

queries(plan_module, opts \\ [])

Every census query for a plan, in the order the plan declares its fields.

script(queries)

The queries as one runnable script, each under a comment saying what it is.

Types

kind()

@type kind() :: :format | :progress | :integrity

Which question a census query answers. See the moduledoc's table.

query()

@type query() :: %{
  kind: kind(),
  schema: module(),
  field: atom(),
  table: String.t(),
  column: String.t(),
  placeholders: [atom()],
  sql: String.t()
}

One rendered query, and enough about it to print a heading over it.

:column is the column the query reads, which for the backfill leg of an adoption migration (into:) is the target column rather than the field the plan names: the census is about what has arrived in the new format.

Functions

header_bytes()

@spec header_bytes() :: pos_integer()

How many leading bytes the format census groups on.

iex> Encryptor.Ecto.Migrator.Census.header_bytes()
4

queries(plan_module, opts \\ [])

@spec queries(
  module(),
  keyword()
) :: [query()]

Every census query for a plan, in the order the plan declares its fields.

Options:

Option	Default
`:prefix`	`nil`	The schema prefix the tables live in

:prefix is here for the same reason verify/2 takes one: a census of a different prefix than the pass wrote to is worse than no census. It is the schema prefix (a PostgreSQL schema), and is unrelated to header_bytes/0's byte prefix, which is a fact about the ciphertext.

script(queries)

@spec script([query()]) :: String.t()

The queries as one runnable script, each under a comment saying what it is.

This is the form an operator is handed: queries/2 is for a caller that wants to render them its own way.