Rotation runbook

Four operator procedures, two of them irreversible. This is the operational half of ADR-0005; read the getting-started guide first if you have not stood a vault up yet.

Every step below is labelled with who performs it:

• [encryptor] - a function this package ships.
• [your store] - an action on a key store this package does not own and cannot see. It defines no table, no migration, no repo and no transaction, and no function here takes one.
• [encryptor_ecto] - the downstream migrator's pass over application rows.
• [operator] - a decision, a deploy, or a wait.

Before anything: the vocabulary

encryptor and encryptor_ecto name the same three levels differently, and the mismatch is dangerous rather than cosmetic. encryptor_ecto calls level 1 "the key-encrypting key" and level 2 "a tenant's data key" - and what this package calls a data key is level 3, per message, generated by the engine, with no row in any table at all.

An operator reading a downstream runbook against this package's names can plausibly run a level 2 rotation believing it is a level 1 rotation. That is the difference between a five-minute pass over thousands of key rows and a multi-hour pass over millions of application rows.

Data key rotation (level 3) is not in this runbook, because it is not an operation an operator performs. It happens on its own every max_age, every max_messages, every max_bytes, and on every cache recycle. The only lever on it is the vault's :cache configuration, and a runbook step for it would be a runbook step for restarting the vault.

R3 is not this package's at all. A change of format, algorithm or encryption context is a re-encrypt that these functions cannot express: rekey/2 preserves the context byte for byte, and the suite is vault configuration rather than a per-call option. It belongs entirely downstream.

What this package ships, and what it only documents

Shipped, as functions:

• Encryptor.Envelope.provision/3 - mints a version (P2 step 1).
• Encryptor.Envelope.rewrap/2 - rewraps one wrapping (P1 step 3).
• Encryptor.Vault.rekey/2 - what rewrap/2 is built on.
• Encryptor.Provider.Static's keys: option - the staged candidate list.

Documented, as procedures: all four below, including every step that touches a store this package does not own.

Not shipped, and deliberately:

• No mix task. Every procedure needs a store this package has no access to, so a task here could only be a task that takes a callback, which is a runbook with a worse interface.
• No scheduler, no expiry, no automatic pruning. An automatic expiry would be a scheduled, unattended, silent data-destruction job.
• No walk over anything. No function here takes a repo, a query, a batch size, or a table.
• No shred/2, no retire/2, no rotate/2. Deleting a wrapping is a DELETE against your store, and wrapping it in a function here would imply this package knows what your store's copies are. It does not.

The window, and what closes it

A rotated-away version keeps decrypting until an operator explicitly deletes its wrapping. This is not a convenience; it is what makes rotation runnable at all. If a version stopped decrypting the moment a newer one became current, re-encrypting a tenant's rows would require stopping writes for the duration of the pass.

The mechanism is already built into the provider contract. encryption_key/2 returns exactly one descriptor and decryption_keys/2 returns a list. Minting version n+1 changes what encryption_key/2 answers; it does not change what decryption_keys/2 answers except by prepending to it. The two are independent by construction, so there is no ordering, no race, and no deploy in which a version becomes uncurrent and undecryptable at the same instant.

Three properties follow, and all three are load-bearing:

• There is no upper bound. This package ships no timer, no expiry column, no TTL, no background sweeper, and no "retire versions older than N" helper. A version left live is live in five years. The window is as long as your policy says, and that policy lives in your compliance documentation, not in this package's configuration.
• There is a lower bound, and it is not zero. A shred is safe only after the rewrite pass has finished, the verification pass is green, and every running vault has stopped serving the retired version out of its materials cache. That last term is the one operators forget; see "Cache drainage" below.
• Widening the window costs a list walk and nothing else. A long candidate list costs one small decrypt per candidate on a cold cache, bounded by the materials cache like every other resolution cost. Nothing about a long window is unsafe. The argument for closing it is that data whose key still exists is data that has not been shredded.

Rotation adds a name; a shred removes one. There is no third mechanism, and pruning is manual. A live set is whatever your store answers decryption_keys/2 with. Rotation calls provision/3 with version: n + 1 and you insert the row; the new version is current because it is the highest, and the old one is live because its row is still there. A shred deletes the row. There is no state between live and gone, no soft-delete this package recognizes, and no retired_at column it reads. Whatever your store's notion of liveness is, membership changes are the only rotation mechanism, and every membership change is an operator action.

Pruning policy is count-based or age-based at your discretion. This package expresses no preference beyond the observation that many live versions cost one small decrypt each on a cold cache, which argues weakly for a count bound. What it does insist on is that pruning is performed by P3 or P4, with their preconditions, and never as a side effect of anything else.

Cache drainage

Deleting a wrapping does not stop a running node from decrypting under it. A vault's materials cache holds resolved materials for up to max_age seconds, so for that long after the delete, every node that has recently served that tenant can still read its data.

Draining has exactly two levers, and both are in every destructive procedure below as an explicit step:

• wait max_age on every vault that serves the affected tenant, or
• restart those vaults.

There is no third option and this package cannot offer one. The engine's cache cannot be enumerated, measured, or selectively invalidated from outside, so a targeted invalidation is not available to be shipped. If the upstream engine grows a bounded, inspectable cache, these steps get sharper.

Cache drainage is part of the procedure, not a follow-up. A shred that stops at the delete is a shred that has not happened yet.

P1. Root rotation (R1, level 1)

Rotates the wrapping subkey. Touches no application data.

Preconditions

• [operator] The reference root is pinned in its own secret. If you followed the getting-started guide, both secrets were provisioned at install holding the same bytes and step 0 below does not apply to you. If your deployment has only one root secret, step 0 applies and it is the most dangerous step in this document.
• [your store] You can enumerate every wrapping for every live version.
• [your store] Updating one wrapping is a single-row write that can be retried.

Steps

0. [operator] Only if this deployment has a single root secret: copy the current root material into a new reference-root secret. Deploy and restart.

Copy it. Do not generate a fresh value. Nothing changes cryptographically; this step exists only so that step 2 does not move the reference. Generating a new value here changes every tenant reference in the deployment, orphaning every stored key name and every header already written, for every tenant. It is recoverable only by restoring the original root material. This is the single most destructive mistake available in this package, and it looks like a no-op.

1. [operator] Generate new root material and place it in the wrapping-root secret, alongside the outgoing value.

2. [operator] Configure the root vault's Static provider with keys: [new, old], newest first, with distinct names. Restart the root vault. Reads now succeed under either subkey; writes use the new one.

   defmodule MyApp.RootVault do
     use Encryptor.Vault, otp_app: :my_app
   
     @impl true
     def init(config) do
       new_root = Base.decode64!(System.fetch_env!("MY_APP_WRAPPING_ROOT_KEY"))
       old_root = Base.decode64!(System.fetch_env!("MY_APP_WRAPPING_ROOT_KEY_PREVIOUS"))
   
       {:ok,
        Keyword.put(config, :provider,
          {Encryptor.Provider.Static,
           keys: [
             [key: Encryptor.Envelope.root_subkey(new_root, "root-wrap"),
              namespace: "encryptor-root", name: "r/v2"],
             [key: Encryptor.Envelope.root_subkey(old_root, "root-wrap"),
              namespace: "encryptor-root", name: "r/v1"]
           ]})}
     end
   end

   The root vault runs cache: false, so the restart costs nothing but the restart. Two entries sharing a name is refused at start: {:invalid_config, :provider, :duplicate_key_names}.

3. [encryptor] + [your store] Walk every live wrapping and call Encryptor.Envelope.rewrap/2 on it, persisting the result:

   for row <- MyApp.MerchantKeys.all_live() do
     {:ok, rewrapped} = Encryptor.Envelope.rewrap(MyApp.RootVault, row)
     MyApp.MerchantKeys.update_wrapping(row, rewrapped)
   end

   Order does not matter and the pass is resumable. rewrap/2 is idempotent in effect and never in bytes: a wrapping already under the new subkey rewraps to different bytes (a fresh data key and a fresh IV go into every message) and an identical descriptor on unwrap. Only the :wrapped field moves; every identity field is carried across unchanged, and so is the binding.

4. [operator] Verify (below). Then remove the outgoing entry from keys: and restart the root vault.

5. [operator] Destroy the outgoing root material from the secret store, once backups taken before step 4 are out of retention or are known to be re-encrypted.

Verification

• Every wrapping's header names the new root key. The wrapping key's name travels in the clear in the encrypted data key's provider info, so this is a census over stored bytes rather than a decrypt pass - Encryptor.Message.describe/1 reads it without a key.
• A sample of tenants resolves: decryption_keys/2 returns descriptors, and a test decrypt of one known ciphertext per sampled tenant succeeds.
• After step 4, the outgoing root material no longer decrypts anything: a deliberate attempt with the old-only configuration in a scratch environment should fail.

Failure and rollback

Reversible at every step until step 5. A pass interrupted at step 3 leaves a mixed population that both configurations can read, which is precisely why step 2 precedes it. To roll back, reverse the order of keys: and re-run step 3; the outgoing subkey becomes current again. The one irreversible action is step 5, and it is separated from the rest by an explicit verification.

P2. Tenant key rotation (R2, level 2)

Rotates one tenant's master key. Every ciphertext for that tenant is rewritten.

The seam runs between the key lifecycle (here) and the row walk (downstream). Steps 1 and 4 are key-store operations and belong to this package; steps 2 and 3 are row operations and belong to encryptor_ecto.

Preconditions

• [your store] The tenant has at least one live version and its current version resolves.
• [operator] The downstream migrator has a plan covering every table and column holding that tenant's ciphertext. A column missed here is a column whose rows become unreadable at step 4. This is the failure mode the whole procedure exists to prevent.
• [operator] Enough time budget to complete the rewrite before the retire. The two are not required to be in the same maintenance session, and they should not be.

Steps

1. [encryptor] + [your store] Mint version n+1; insert the row. Version n+1 is now current for new writes; n remains live. The window opens here.

   {:ok, wrapped} =
     Encryptor.Envelope.provision(MyApp.RootVault, merchant.id,
       reference_subkey: reference_subkey,
       namespace: "acme-merchant",
       version: 2
     )
   
   {:ok, _row} = MyApp.MerchantKeys.insert(wrapped)

   Both versions now resolve. New writes take v2; old rows still open under v1:

   {:ok, [%Encryptor.Key.Aes{name: "t/" <> _ = v2}, %Encryptor.Key.Aes{name: v1}]} =
     MyApp.MerchantKeyProvider.decryption_keys(state, merchant.id)
   # v2 is "t/<tenant_ref>/v2", v1 is "t/<tenant_ref>/v1", newest first.

2. [encryptor_ecto] Run the migrator over the tenant's scope. Nothing in this package participates. Every row it rewrites is re-encrypted under whatever encryption_key/2 now answers, which is n+1, without the migrator naming a version.

3. [encryptor_ecto] Run the verification pass over the same scope - the whole scope, not a sample.

4. [operator] Only when step 3 is green, run P4 for version n.

Days may pass between steps 1 and 4, and the window stays open the whole time. Nothing forces step 4 to ever run.

Verification

• The migrator's verifier exits zero over the full scope.
• No row remains whose header names version n, per the same census as P1.
• The tenant's decryption_keys/2 still contains both versions at this point. If it does not, something removed a version outside this procedure and P4 must not be run until that is understood.

Failure and rollback

Fully reversible until step 4. An interrupted rewrite leaves rows under both versions, all readable, and the pass resumes. Rolling back the rotation itself means making n current again, which is a question your store answers, and the already-rewritten rows stay readable either way because both versions are live. There is no state in this procedure, before step 4, from which data can be lost.

P3. Tenant shred (R4, offboarding)

Destroys every version of one tenant's master key. Irreversible.

Preconditions

• [operator] A recorded, human decision that this tenant's data is to be destroyed, with a reference to it in the change record. A shred is never automated and never a cascade from a DELETE on a tenants table.
• [operator] Every legitimate consumer of that tenant's data has been identified. After this procedure, no amount of key material recovers it.
• [operator] The backup implications below have been read and accepted.

Steps

1. [your store] Confirm the tenant reference resolves and enumerate the wrappings that are about to be destroyed. Record the count and the version numbers in the change record.
2. [your store] Delete every wrapping for the tenant.
3. [operator] Drain the caches: wait max_age on every vault that serves the tenant, or restart those vaults. Until this completes, a running node can still decrypt the tenant's data from cached materials.
4. [your store] Delete the tenant's ciphertext rows. See "What a shred does not destroy" below: whether this step is optional depends on whether tenant attribution is itself personal data in your jurisdiction.

Verification

• encrypt/2 and decrypt/2 for that tenant return {:error, %Encryptor.Error{reason: {:unknown_key, selector}}}, not :decrypt_failed. This is the acceptance test for the shred.
• The key store returns no rows for the tenant reference.
• A ciphertext known to belong to the tenant does not decrypt from any node.

Failure and rollback

None after step 2. A partially completed step 2 leaves some versions live and some destroyed - some rows readable, some permanently not, and the two indistinguishable to the application. That is the worst state in this document. If step 2 fails partway, complete it rather than reverting it, and record what was destroyed.

P4. Version retire (R4, one version)

Closes the window opened by P2. Irreversible.

It is a separate procedure from P2 on purpose, so that it is a separate decision with its own preconditions rather than the last line of a longer procedure.

Preconditions

• [operator] P2 steps 2 and 3 completed and green, over the whole scope, not a sample. This is the fence. There is no legitimate reason to retire a version whose rows have not been verifiably rewritten.
• [operator] The verification is recent enough that no traffic since could have written under the retired version. It cannot have - encryption_key/2 has answered n+1 since P2 step 1 - so this precondition is about your confidence in the scope, not about the mechanism.
• [operator] Cache drainage is understood to be part of this procedure, not a follow-up.

Steps

1. [your store] Delete the wrapping for version n.
2. [operator] Drain the caches, as P3 step 3.

Verification

• decryption_keys/2 for the tenant no longer contains version n.
• The application's error rate for that tenant is unchanged. A rise in :decrypt_failed means rows were missed, and they are now unrecoverable.

Failure and rollback

None.

The two failure shapes after a destruction

# After P4, on a row the plan missed. Indistinguishable from corruption:
MyApp.MerchantVault.decrypt(missed_row, key: merchant.id, encryption_context: ctx)
#=> {:error, %Encryptor.Error{reason: :decrypt_failed, operation: :decrypt}}

# After P3, the whole tenant. Specific, and not an oracle:
MyApp.MerchantVault.decrypt(any_row, key: shredded.id, encryption_context: ctx)
#=> {:error, %Encryptor.Error{reason: {:unknown_key, "..."}, operation: :decrypt}}

A whole-tenant shred is loudly distinguishable from corruption: the provider finds no live row and fails at resolution time, before any ciphertext is examined, on a term that depends only on the caller's selector.

A single retired version is not distinguishable. A message written under a retired version fails as :decrypt_failed, identical to a corrupted message, a wrong-key message, and a context mismatch, because the whole decrypt-side failure space collapses to one reason and this runbook does not reopen that.

The operational answer is the fence, not a new error term. P4's precondition is a green whole-scope verification, so a :decrypt_failed after a retire means a row the plan did not cover - a bug in the plan, not an ambiguity in the error.

What a shred does not destroy

Read this before promising anyone that your offboarding crypto-shreds.

A shred destroys plaintext, not attribution. Every message header carries the tenant's permanent pseudonym - the tenant_ref, in the encrypted data key's name and in the encryption context - and deleting the wrapping does not touch it. Outsiders cannot resolve the pseudonym. The holder of the reference subkey can, by guess-and-confirm, forever, in every retained backup: derive the reference for a candidate tenant id, compare it against the header, and the match is a confirmation. The reference subkey is never rotated, so this does not expire.

Consequently, P3 step 4's deletion of the tenant's ciphertext rows is compliance-mandatory wherever tenant attribution is itself personal data, not hygiene. And the shred claim must never be stated as full erasure.

A shred is only as good as the copies. Deleting a wrapping destroys the key in the primary store. It does nothing about:

• database backups taken while the wrapping existed,
• read replicas that have not yet received the delete,
• a logical dump on someone's laptop,
• a wrapping exported into a disaster-recovery vault,
• a snapshot of a running node's memory.

A host whose compliance story is "we crypto-shred on offboarding" has committed to a backup retention and replica story this package cannot see and does not verify. Stating the limit is the most it can honestly do.

A shred is wider than the columns this package encrypted. A tenant master key is also the derivation root for any purpose-labelled subkey - a blind index, a search key - and those subkeys are recomputed on demand and never stored. They die with the master key. That is intended, and it means a shred reaches further than an inventory of encrypted columns would suggest.

Blast radius: what an attacker holding each combination can read

This is the trust boundary the design is shaped around.

Four things follow, and each is a limitation as much as a property:

• The two-factor property is real but conditional. Separating the root key (environment, secrets manager) from the wrapped keys (database) means a database compromise alone - the overwhelmingly common one, from a backup, a read replica, a dump, an SQL injection - yields nothing. This is the design's main return.
• The running process is not protected against, at all. A process that can encrypt for a tenant necessarily holds that tenant's master key in memory, and one that can provision holds the root. Any attacker with code execution or memory read in the application defeats the hierarchy entirely. Moving the root to a key manager narrows this - the root material stops being in BEAM memory - but the unwrapped tenant keys still are.
• Tenant compromise is bounded by version, not by time. One master key covers every message written while it was current. A tenant compromised at version 3 exposes exactly the data encrypted under version 3, which is why rotation cadence is a real security parameter and not hygiene theatre.
• Shredding is only as good as the copies. See above.

Blast radius: what each step destroys

"Reversible" means reversible by an operator holding everything they held before the step, without recourse to backups.

The division of labour with encryptor_ecto

Stated from this side, so both packages' documentation agrees:

• The Ecto layer resolves a tenant and passes it as key:. It does not put a tenant pair in :encryption_context - that pair is refused from a caller - and its missing-tenant error fires before this package is called at all.
• The Ecto layer supplies table and column from its frozen declared values. Renaming a physical table or column while keeping the declared value pinned does not invalidate stored rows; changing the declared value is an R3 re-encrypt, not a rotation.
• The Ecto layer enforces nothing. What makes those keys required is your vault's required_context: ["table", "column"]. Enforcement lives on the vault because it is a property of the vault, not of one type module: two schemas sharing a vault must not be able to disagree about how strictly their rows are bound.
• P1 step 3's walk over live wrappings and P3 step 2's delete are operations on the wrapped-key table, which lives in encryptor_ecto. They are not migrator operations: they take no plan, no checkpoint, no compare-and-swap and no batch, and they never touch a ciphertext column of an application table.

Cadence

This package gives no numbers and no default, consistent with declining to ship a scheduler. What it will say is the argument: tenant compromise is bounded by version rather than by time, so cadence is a security parameter with a reason attached rather than hygiene, and the right interval is a function of how much data one version is allowed to cover.

Records

ADR-0005 (rotation and crypto-shred) owns every procedure here; ADR-0003 decision 10 owns the attacker table; ADR-0004 decision 10 owns the division of labour. Where this guide and a record disagree, the record wins and the disagreement is a bug in this guide.