reckon_db_store_healer (reckon_db v5.8.0)

View Source

Store healer for reckon-db — continuous self-healing of a store's Ra cluster.

One healer per store (cluster mode only). It closes the loop that the coordinator + consistency facade leave open: detection existed, but nothing took corrective action, so a replica that boots into its own singleton (the classic deploy join-race) or drifts out of quorum stays orphaned until an operator runs a wipe-and-rejoin script by hand.

The healer runs an always-armed periodic audit. Each tick it asks the authoritative, reachability-based facade [[reckon_db_cluster]] whether THIS replica is in a healthy, quorum-holding cluster. If not, it looks for a majority cluster (one with an elected leader — which by Raft necessarily holds quorum) that this replica is NOT part of, and, if the data-safety gate passes, resets the local diverged state and rejoins.

Data-safety gate

A reset is destructive (it discards the local Ra/Khepri state), so safe_to_reset/1 permits it ONLY when ALL hold: 1. a majority cluster with an elected leader exists among the peers, 2. this node is NOT that leader, and 3. this node is NOT a member of that majority cluster.

(3) is the key invariant: we only ever reset a replica the majority has never accepted (the never-joined singleton) — never a member of the authoritative cluster, and never the majority itself. A transient network partition of a real member is left to Ra to heal on its own.

Set self_heal => alarm_only in the store config options to detect and emit telemetry without ever taking destructive action.

Summary

Types

index_decl/0
integrity_config/0
integrity_key_source/0
store_config/0

Functions

audit_now(StoreId)

Force an immediate audit (for tests / operators).

handle_call(Request, From, State)
handle_cast(Msg, State)
handle_info(Info, State)
init(Store_config)
start_link(Store_config)
status(StoreId)

Self-healing status for the store — surfaced by reckon_db_cluster:health_check/1 for the admin dashboard.

terminate(Reason, State)

Types

index_decl/0

-type index_decl() ::
          tags | event_type |
          {meta, Key :: binary()} |
          {payload, Key :: binary()} |
          {payload_hash, Keys :: [binary()]}.

integrity_config/0

-type integrity_config() :: disabled | #{enabled := true, key_source := integrity_key_source()}.

integrity_key_source/0

-type integrity_key_source() :: {env_var, EnvName :: binary()} | {sealed_file, Path :: file:filename()}.

store_config/0

-type store_config() ::
          #store_config{store_id :: atom(),
                        data_dir :: string(),
                        mode :: single | cluster,
                        timeout :: pos_integer(),
                        writer_pool_size :: pos_integer(),
                        reader_pool_size :: pos_integer(),
                        gateway_pool_size :: pos_integer(),
                        options :: map(),
                        integrity :: integrity_config(),
                        indexes :: [index_decl()]}.

Functions

audit_now(StoreId)

-spec audit_now(atom()) -> ok.

Force an immediate audit (for tests / operators).

handle_call(Request, From, State)

handle_cast(Msg, State)

handle_info(Info, State)

init(Store_config)

start_link(Store_config)

-spec start_link(store_config()) -> {ok, pid()} | {error, term()}.

status(StoreId)

-spec status(atom()) -> #{atom() => term()}.

Self-healing status for the store — surfaced by reckon_db_cluster:health_check/1 for the admin dashboard.

Reads a persistent_term the healer publishes on every audit. A plain term read (no gen_server call) is essential: health_check/1 is called by both the coordinator and the healer itself while they hold each other's calls, so routing this through the healer's mailbox would deadlock.

terminate(Reason, State)