Bedrock.ControlPlane.Director.Recovery (bedrock v0.5.2)

View Source

Orchestrates distributed system recovery through a coordinated phase sequence.

This module implements Bedrock's recovery orchestration, which rebuilds the transaction system after critical component failures. Recovery follows a linear state machine where each phase either transitions to the next phase or stalls pending resource availability.

The process begins by attempting to lock services from the previous transaction system layout, then branches into either first-time initialization or recovery from existing persistent state. Each phase validates its prerequisites and may stall if conditions are not met, with retry logic triggered when the environment changes.

Recovery attempts are persisted at major milestones, allowing resumption from consistent checkpoints if interrupted. The orchestrator coordinates between phases but delegates specific recovery logic to individual phase modules.

Critical components that trigger recovery include coordinators, directors, sequencers, commit proxies, resolvers, and transaction logs. Storage servers and gateways handle failures independently without triggering full recovery.

See Bedrock.ControlPlane.Director for epoch management and Bedrock.ControlPlane.Director.Nodes for service discovery integration.

Summary

Types

recovery_context()

@type recovery_context() :: %{
  cluster_config: Bedrock.ControlPlane.Config.t(),
  old_transaction_system_layout:
    Bedrock.ControlPlane.Config.TransactionSystemLayout.t(),
  node_capabilities: %{required(Bedrock.Cluster.capability()) => [node()]},
  lock_token: binary(),
  available_services: %{
    required(Bedrock.Service.Worker.id()) => {atom(), {atom(), node()}}
  },
  coordinator: pid()
}

Functions

do_recovery(t)

persist_config(t)

persist_new_transaction_system_layout(t)

@spec persist_new_transaction_system_layout(Bedrock.ControlPlane.Director.State.t()) ::
  Bedrock.ControlPlane.Director.State.t()

run_recovery_attempt(t, context, next_phase_module \\ __MODULE__.TSLValidationPhase)

setup_for_initial_recovery(t)

setup_for_subsequent_recovery(t)

try_to_recover(t)