reckon_db_store_healer (reckon_db v5.8.1)

View Source

Store healer for reckon-db — continuous self-healing of a store's Ra cluster.

One healer per store (cluster mode only). It closes the loop that the coordinator + consistency facade leave open: detection existed, but nothing took corrective action, so a replica that boots into its own singleton (the classic deploy join-race) or drifts out of quorum stays orphaned until an operator runs a wipe-and-rejoin script by hand.

The healer runs an always-armed periodic audit. Each tick it asks the authoritative, reachability-based facade [[reckon_db_cluster]] whether THIS replica is in a healthy, quorum-holding cluster. If not, it looks for a majority cluster (one with an elected leader — which by Raft necessarily holds quorum) that this replica is NOT part of, and, if the data-safety gate passes, resets the local diverged state and rejoins.

Data-safety gate

A reset is destructive (it discards the local Ra/Khepri state), so safe_to_reset/1 permits it ONLY when ALL hold: 1. a majority cluster with an elected leader exists among the peers, 2. this node is NOT that leader, and 3. this node is NOT locally clustered with that leader (the majority's leader is absent from OUR OWN local member set).

(3) is the key invariant, and it keys on our LOCAL view, not the majority's: a node that reset into its own singleton has local members = [self] (leader absent) and is healed; a node merely partitioned from a cluster it is still configured in keeps the full local member set (leader present) and is left to Ra. We never reset the majority itself, and never a node still locally clustered with the authoritative leader.

Set self_heal => alarm_only in the store config options to detect and emit telemetry without ever taking destructive action.

Summary

Types

index_decl/0
integrity_config/0
integrity_key_source/0
store_config/0

Functions

audit_now(StoreId)

Force an immediate audit (for tests / operators).

handle_call(Request, From, State)
handle_cast(Msg, State)
handle_info(Info, State)
init(Store_config)
start_link(Store_config)
status(StoreId)

Self-healing status for the store — surfaced by reckon_db_cluster:health_check/1 for the admin dashboard.

terminate(Reason, State)

Types

index_decl/0

-type index_decl() ::
          tags | event_type |
          {meta, Key :: binary()} |
          {payload, Key :: binary()} |
          {payload_hash, Keys :: [binary()]}.

integrity_config/0

-type integrity_config() :: disabled | #{enabled := true, key_source := integrity_key_source()}.

integrity_key_source/0

-type integrity_key_source() :: {env_var, EnvName :: binary()} | {sealed_file, Path :: file:filename()}.

store_config/0

-type store_config() ::
          #store_config{store_id :: atom(),
                        data_dir :: string(),
                        mode :: single | cluster,
                        timeout :: pos_integer(),
                        writer_pool_size :: pos_integer(),
                        reader_pool_size :: pos_integer(),
                        gateway_pool_size :: pos_integer(),
                        options :: map(),
                        integrity :: integrity_config(),
                        indexes :: [index_decl()]}.

Functions

audit_now(StoreId)

-spec audit_now(atom()) -> ok.

Force an immediate audit (for tests / operators).

handle_call(Request, From, State)

handle_cast(Msg, State)

handle_info(Info, State)

init(Store_config)

start_link(Store_config)

-spec start_link(store_config()) -> {ok, pid()} | {error, term()}.

status(StoreId)

-spec status(atom()) -> #{atom() => term()}.

Self-healing status for the store — surfaced by reckon_db_cluster:health_check/1 for the admin dashboard.

Reads a persistent_term the healer publishes on every audit. A plain term read (no gen_server call) is essential: health_check/1 is called by both the coordinator and the healer itself while they hold each other's calls, so routing this through the healer's mailbox would deadlock.

terminate(Reason, State)