reckon_db_store_healer (reckon_db v5.8.2)

View Source

Store healer for reckon-db — continuous self-healing of a store's Ra cluster.

One healer per store (cluster mode only). It closes the loop that the coordinator + consistency facade leave open: detection existed, but nothing took corrective action, so a replica that boots into its own singleton (the classic deploy join-race) or drifts out of quorum stays orphaned until an operator runs a wipe-and-rejoin script by hand.

The healer runs an always-armed periodic audit. Each tick it asks the authoritative, reachability-based facade [[reckon_db_cluster]] whether THIS replica is in a healthy, quorum-holding cluster. If not, it looks for a majority cluster (one with an elected leader — which by Raft necessarily holds quorum) that this replica is NOT part of, and, if the data-safety gate passes, resets the local diverged state and rejoins.

Data-safety gate

A reset is destructive (it discards the local Ra/Khepri state), so safe_to_reset/1 (== is_orphan/1) permits it ONLY when: 1. a majority cluster with an elected leader exists among the peers, 2. this node is NOT that leader, and 3. EITHER this node is not locally clustered with that leader (diverged into its own cluster) OR its local ra server is wedged (local_responsive => false).

All probes are lock-free (ra_leaderboard ETS) or hard timeout-bounded; the audit must NEVER call khepri_cluster:members / get_quorum_status, which do an unbounded statem_call into the local ra server — the very server that is wedged in the case we most need to heal. A monitor that calls the monitored server's synchronous API inherits its wedge. A node still clustered with the leader AND responsive is a transient partition, left to Ra; we never reset the majority itself.

Set self_heal => alarm_only in the store config options to detect and emit telemetry without ever taking destructive action.

Summary

Types

index_decl/0
integrity_config/0
integrity_key_source/0
store_config/0

Functions

audit_now(StoreId)

Force an immediate audit (for tests / operators).

handle_call(Request, From, State)
handle_cast(Msg, State)
handle_info(Info, State)
init(Store_config)
start_link(Store_config)
status(StoreId)

Self-healing status for the store — surfaced by reckon_db_cluster:health_check/1 for the admin dashboard.

terminate(Reason, State)

Types

index_decl/0

-type index_decl() ::
          tags | event_type |
          {meta, Key :: binary()} |
          {payload, Key :: binary()} |
          {payload_hash, Keys :: [binary()]}.

integrity_config/0

-type integrity_config() :: disabled | #{enabled := true, key_source := integrity_key_source()}.

integrity_key_source/0

-type integrity_key_source() :: {env_var, EnvName :: binary()} | {sealed_file, Path :: file:filename()}.

store_config/0

-type store_config() ::
          #store_config{store_id :: atom(),
                        data_dir :: string(),
                        mode :: single | cluster,
                        timeout :: pos_integer(),
                        writer_pool_size :: pos_integer(),
                        reader_pool_size :: pos_integer(),
                        gateway_pool_size :: pos_integer(),
                        options :: map(),
                        integrity :: integrity_config(),
                        indexes :: [index_decl()]}.

Functions

audit_now(StoreId)

-spec audit_now(atom()) -> ok.

Force an immediate audit (for tests / operators).

handle_call(Request, From, State)

handle_cast(Msg, State)

handle_info(Info, State)

init(Store_config)

start_link(Store_config)

-spec start_link(store_config()) -> {ok, pid()} | {error, term()}.

status(StoreId)

-spec status(atom()) -> #{atom() => term()}.

Self-healing status for the store — surfaced by reckon_db_cluster:health_check/1 for the admin dashboard.

Reads a persistent_term the healer publishes on every audit. A plain term read (no gen_server call) is essential: health_check/1 is called by both the coordinator and the healer itself while they hold each other's calls, so routing this through the healer's mailbox would deadlock.

terminate(Reason, State)