Bedrock.ControlPlane.Director.Recovery.TopologyPhase (bedrock v0.5.3)
View SourceConstructs the Transaction System Layout (TSL), the blueprint defining how all components in the recovered system connect and communicate.
The TSL solves the coordination problem in distributed systems: without it, individual components exist in isolation unable to find each other. This phase validates component availability, builds the complete topology, and prepares services for the system transaction.
Validation Process
Validates core transaction components:
- Sequencer: Must have valid process ID (exactly one runs cluster-wide)
- Commit Proxies: Must have valid process IDs for all proxies (list cannot be empty)
- Resolvers: Must have valid
{start_key, process_id}pairs defining key range responsibilities - Logs: Must have corresponding service entries with
{:up, process_id}status
TSL Construction
Builds the complete TSL data structure containing component process IDs, service mappings, metadata materializer reference, and shard layout.
Service Mapping Algorithm
The services field maps service IDs to descriptors through:
- Extract all service IDs from logs
- For each service ID, check existence in context.available_services
- Build service descriptors containing:
- kind: Service type (log, etc.)
- last_seen: Timestamp from service discovery
- status: Either
{:up, process_id}or:down
Selective Service Unlocking
Commit proxies are unlocked via CommitProxy.recover_from/5 with lock token and TSL.
Other components (sequencer, resolvers, logs) transition automatically when
system transaction completes.
Error Handling
- Validation failures:
{:stalled, {:recovery_system_failed, {:invalid_recovery_state, reason}}} - Construction failures:
{:stalled, {:recovery_system_failed, reason}} - Unlock failures:
{:stalled, {:recovery_system_failed, {:unlock_failed, reason}}}
Transitions to PersistencePhase to commit layout through system transaction.
Summary
Functions
Executes TSL construction with validation, building, and unlocking phases.
Functions
Executes TSL construction with validation, building, and unlocking phases.
This function implements the three-step TSL construction process:
- Validation: Confirms all transaction components are operational via
validate_recovery_state/1 - Construction: Builds the complete TSL data structure via
build_transaction_system_layout/2 - Unlocking: Selectively unlocks commit proxies and storage servers via
unlock_services/4
Parameters
recovery_attempt- Recovery state containing component process IDs and configurationcontext- Recovery context with available services, lock token, and capabilities
Returns
{updated_recovery_attempt, PersistencePhase}- Success with TSL added to recovery attempt{recovery_attempt, {:stalled, {:recovery_system_failed, reason}}}- Failure with specific error
Process Flow
Uses with to ensure all steps succeed before transitioning. Any failure at validation,
construction, or unlocking causes recovery to stall with detailed error information.
The updated recovery attempt includes the constructed TSL, enabling the persistence phase to commit the layout through a system transaction.