Bedrock.ControlPlane.Director.Recovery.TopologyPhase (bedrock v0.5.2)

View Source

Constructs the Transaction System Layout (TSL), the blueprint defining how all components in the recovered system connect and communicate.

The TSL solves the coordination problem in distributed systems: without it, individual components exist in isolation unable to find each other. This phase validates component availability, builds the complete topology, and prepares services for the system transaction.

Validation Process

Validates core transaction components:

  • Sequencer: Must have valid process ID (exactly one runs cluster-wide)
  • Commit Proxies: Must have valid process IDs for all proxies (list cannot be empty)
  • Resolvers: Must have valid {start_key, process_id} pairs defining key range responsibilities
  • Logs: Must have corresponding service entries with {:up, process_id} status

TSL Construction

Builds the complete TSL data structure containing component process IDs, service mappings, metadata materializer reference, and shard layout.

Service Mapping Algorithm

The services field maps service IDs to descriptors through:

  1. Extract all service IDs from logs
  2. For each service ID, check existence in context.available_services
  3. Build service descriptors containing:
    • kind: Service type (log, etc.)
    • last_seen: Timestamp from service discovery
    • status: Either {:up, process_id} or :down

Selective Service Unlocking

Commit proxies are unlocked via CommitProxy.recover_from/5 with lock token and TSL. Other components (sequencer, resolvers, logs) transition automatically when system transaction completes.

Error Handling

  • Validation failures: {:stalled, {:recovery_system_failed, {:invalid_recovery_state, reason}}}
  • Construction failures: {:stalled, {:recovery_system_failed, reason}}
  • Unlock failures: {:stalled, {:recovery_system_failed, {:unlock_failed, reason}}}

Transitions to PersistencePhase to commit layout through system transaction.

Summary

Functions

Executes TSL construction with validation, building, and unlocking phases.

Functions

execute(recovery_attempt, context)

Executes TSL construction with validation, building, and unlocking phases.

This function implements the three-step TSL construction process:

  1. Validation: Confirms all transaction components are operational via validate_recovery_state/1
  2. Construction: Builds the complete TSL data structure via build_transaction_system_layout/2
  3. Unlocking: Selectively unlocks commit proxies and storage servers via unlock_services/4

Parameters

  • recovery_attempt - Recovery state containing component process IDs and configuration
  • context - Recovery context with available services, lock token, and capabilities

Returns

  • {updated_recovery_attempt, PersistencePhase} - Success with TSL added to recovery attempt
  • {recovery_attempt, {:stalled, {:recovery_system_failed, reason}}} - Failure with specific error

Process Flow

Uses with to ensure all steps succeed before transitioning. Any failure at validation, construction, or unlocking causes recovery to stall with detailed error information.

The updated recovery attempt includes the constructed TSL, enabling the persistence phase to commit the layout through a system transaction.