Bedrock.ControlPlane.Director.Recovery (bedrock v0.6.0)

View Source

Orchestrates distributed system recovery through a coordinated phase sequence.

This module implements Bedrock's recovery orchestration, which rebuilds the transaction system after critical component failures. Recovery follows a linear state machine where each phase either transitions to the next phase or stalls pending resource availability.

The process begins by attempting to lock services from the previous transaction system layout, then branches into either first-time initialization or recovery from existing persistent state. Each phase validates its prerequisites and may stall if conditions are not met, with retry logic triggered when the environment changes.

Recovery attempts are persisted at major milestones, allowing resumption from consistent checkpoints if interrupted. The orchestrator coordinates between phases but delegates specific recovery logic to individual phase modules.

Critical components that trigger recovery include coordinators, directors, sequencers, commit proxies, resolvers, and transaction logs. Storage servers and gateways handle failures independently without triggering full recovery.

See Bedrock.ControlPlane.Director for epoch management and Bedrock.ControlPlane.Director.Nodes for service discovery integration.

Summary

Types

recovery_context()

@type recovery_context() :: %{
  cluster_config: Bedrock.ControlPlane.Config.t(),
  old_transaction_system_layout:
    Bedrock.ControlPlane.Config.TransactionSystemLayout.t(),
  node_capabilities: %{required(Bedrock.Cluster.capability()) => [node()]},
  lock_token: binary(),
  available_services: %{
    required(Bedrock.Service.Worker.id()) => {atom(), {atom(), node()}}
  },
  coordinator: pid()
}

Functions

do_recovery(t)

ghost_directory_ids(services, arg2)

@spec ghost_directory_ids(
  services :: %{required(Bedrock.Service.Worker.id()) => term()},
  Bedrock.ControlPlane.Config.TransactionSystemLayout.t()
) :: [Bedrock.Service.Worker.id()]

The directory ids a completed recovery's layout does not reference.

These are ghosts: registrations left behind by workers on nodes that no longer exist under that name (node names change across restarts, and nothing on a dead node can deregister itself). Entries on live nodes need no help here — their foreman retires and deregisters them through layout reconciliation — but only the director can clean up for the dead.

persist_config(t)

persist_new_transaction_system_layout(t)

@spec persist_new_transaction_system_layout(Bedrock.ControlPlane.Director.State.t()) ::
  Bedrock.ControlPlane.Director.State.t()

run_recovery_attempt(t, context, next_phase_module \\ __MODULE__.TSLValidationPhase)

setup_for_initial_recovery(t)

setup_for_subsequent_recovery(t)

try_to_recover(t)