What a finished execution leaves behind

An execution that has ended keeps three things in your store: its row in the executions table, the last position blob written on that row, and, if your adapter keeps one, its whole input log. The row is history. The position blob and the log are not needed by anything in this package once the execution has ended, and the log holds your own event data, verbatim.

This page says what StatifierPersistence.Retention.prune/3 clears, which rows you may delete yourself for a finished execution, and which you must not. The schema declares no foreign keys, so the database will not stop a wrong delete. This page is the only guard.

The design is ADR-0016.

Pruning

cutoff = DateTime.add(DateTime.utc_now(), -90, :day)

{:ok, %{executions: pruned, position_blobs: blobs, inputs: inputs}} =
  StatifierPersistence.Retention.prune(store, cutoff)

prune/3 selects every execution whose status is :completed, :failed or :cancelled and whose ended_at is before cutoff. For each one it sets position_blob to NULL and deletes all of its input log rows. The row itself stays, with its status, failure, metadata, answer and ended_at.

• The cutoff is yours. It is a DateTime, and prune/3 raises on a number of days, a Duration or a Date. This package has no retention window and no default. How long you keep a finished execution's leftovers is your policy, and you run the prune when that policy says to.
• An execution with no ended_at is never pruned. That includes a row that was already finished before migration V08 added the column: its end time is not stored anywhere. It gets a stamp only when a later terminal write reaches it.
• A stamped row that is not terminal is never pruned. A stamp stays on a row that a later write put back to :active, and that execution can still take a step.
• It works in batches and can be repeated. batch_size: (default 500) bounds each transaction. Each batch commits on its own, so a call that fails part-way leaves the earlier batches done. Calling again with the same cutoff carries on, and a call with nothing left to do answers zeros. This holds when nothing encloses the call; see "Pruning inside your own transaction" below for what changes when something does.
• It needs an adapter that declares it. Both shipped adapters do. The in-memory adapter keeps no input log, so it clears position blobs only. An adapter of your own declares it by exporting supports_execution_pruning?/1 and prune_executions/4; one that does not gets {:error, :execution_pruning_unsupported}, and StatifierPersistence.Storage.execution_pruning_supported?/1 tells you in advance.

Pruning one partition

If your tables carry a column of your own that partitions them, placed with :leading_columns on use StatifierPersistence.Ecto, scope: confines a prune to the rows that hold it:

StatifierPersistence.Retention.prune(store, cutoff, scope: [tenant_id: tenant_id])

scope: is a keyword list of column equalities. Every statement each batch runs carries all of them: the selection, the input log check inside it, the input log delete and the position blob update. So a prune you run inside one partition's transaction reads and writes no row of another.

• The columns must be your :leading_columns. The Ecto adapter raises ArgumentError for any other column before it runs a statement.
• A scope is never empty and never nil. prune/3 raises on scope: [], so a scope your code computed to nothing cannot prune every partition. It also raises on a nil value, because an equality with NULL matches no row. To prune across every partition, leave scope: out.
• The in-memory adapter cannot scope. Its records have no columns of yours, so it answers {:error, :unscoped_adapter} for any scope: and clears nothing.

Pruning inside your own transaction

prune/3 called with nothing enclosing it commits each batch on its own. Called inside a transaction of your own, its whole drain - every batch - runs as that one transaction, and commits or rolls back with it. batch_size: then bounds each batch's statements, not the transaction, so a host that opens a transaction to set a partition's context before any write and calls prune/3 there loses the short-transaction bound batch_size: exists for.

For one bounded transaction per batch, call prune/3 with single_batch: true inside each of your own transactions, and call again while more? is true:

def prune_tenant(store, cutoff, tenant_id) do
  batch =
    Repo.transaction(fn ->
      # set the tenant's context for this transaction here, then:
      opts = [scope: [tenant_id: tenant_id], single_batch: true]

      case StatifierPersistence.Retention.prune(store, cutoff, opts) do
        {:ok, counts} -> counts
        {:error, reason} -> Repo.rollback(reason)
      end
    end)

  with {:ok, %{more?: true}} <- batch do
    prune_tenant(store, cutoff, tenant_id)
  end
end

It answers {:ok, counts} for the last batch, with more?: false, or the first {:error, reason}. Every batch before that one has committed, so calling it again carries on.

more? is true when the batch took as many executions as batch_size: allows, so another call may find more, and false when it took fewer - the same point where the default drain stops on its own. When the last due batch holds exactly batch_size: executions, that call answers more?: true and the next one answers zeros with more?: false, so a loop that stops at more?: false makes one extra call that clears nothing. A row another transaction holds locked is skipped, not waited on, so it is left for a later call either way.

After a prune, Executions.inputs/2 answers {:ok, []} for that execution, which is also what an execution that took no input answers. A load of its position answers {:error, :execution_position_missing}, which is also what an execution that failed at creation answers. No read tells you an execution was pruned. If something you run reads input logs, for example a replay tool, decide which executions it may read from their ended_at and your own cutoff.

Rows you may delete for a finished execution

"Finished" here means the same thing it means to prune/3: the status is :completed, :failed or :cancelled, and it has an ended_at.

Rows you must not delete

What pruning does not do

• It keeps the answer and the metadata. outcome_blob, failure and metadata stay on the row, because a parent may still need the answer. Metadata should hold host identities only, never personal data (see ADR-0006). If your chart's answers carry personal data, pruning does not remove it.
• It does not touch keys or copies. If you encrypt the blob columns (:blob_type), pruning removes the ciphertext from the live rows. It does not rotate or destroy a key, and it does not reach your backups.
• It does not cascade. A parent and its durable children are each pruned on their own ended_at. Either order is safe, because the rows stay.
• It has nothing to clear from the trace. This package emits telemetry and stores none of it.