Ferricstore.Store.Promotion (ferricstore v0.4.2)

Copy Markdown View Source

Collection promotion: migrates large compound-key collections from the shared shard Bitcask into a dedicated per-key Bitcask instance.

Background (spec section 2B.4b)

Small collections (hashes, sets, sorted sets) are stored as compound keys in the shared shard Bitcask (H:key\0field, S:key\0member, Z:key\0member). When any compound-key collection exceeds the configurable promotion threshold (default: 100 entries), it is promoted to a dedicated Bitcask instance stored under:

dedicated/shard_N/{type}:{sha256_of_key}/

where {type} is hash, set, or zset.

Promotion is one-way -- once promoted, a collection stays in its dedicated instance even if entries are later deleted below the threshold. The dedicated instance is only removed when the entire key is deleted via DEL / UNLINK.

Lists are not promoted

Lists store all elements as a single serialized Erlang term in one Bitcask entry (via ListOps). Since there is no compound key fan-out, a list with 1000 elements is still a single Bitcask entry and does not benefit from promotion. List promotion is intentionally skipped.

Promotion marker

When a key is promoted, a marker entry PM:redis_key is written to the shared Bitcask with the type as its value ("hash", "set", or "zset"). This allows the shard to rediscover promoted keys on restart by scanning for PM: prefixed keys during initialization.

Configuration

config :ferricstore, :promotion_threshold, 100

Set to 0 to disable automatic promotion entirely (no collections will ever be promoted).

Summary

Functions

await_compaction_latch(owner, redis_key)

Waits until no promoted compaction latch is held for redis_key.

cleanup_promoted!(redis_key, shard_data_path, keydir, data_dir, shard_index, instance_ctx \\ nil)
dedicated_path(data_dir, shard_index, type, redis_key)
find_active(path)

Returns the active (highest file_id) log file path in a dedicated directory.

marker_key(redis_key)
open_dedicated(data_dir, shard_index, type, redis_key)
promote_collection!(type, redis_key, shard_data_path, keydir, data_dir, shard_index, instance_ctx \\ nil)
promote_hash!(redis_key, shared_store, keydir, data_dir, shard_index, instance_ctx \\ nil)
recover_promoted(shard_data_path, keydir, data_dir, shard_index, instance_ctx \\ nil)
threshold()
threshold(ctx)

Returns the promotion threshold from instance ctx.

with_compaction_latch(owner, redis_key, fun)

Runs fun while holding the per-promoted-key compaction latch.

Functions

await_compaction_latch(owner, redis_key)

@spec await_compaction_latch(map(), binary()) :: :ok

Waits until no promoted compaction latch is held for redis_key.

Called by Raft apply before appending promoted dedicated writes/tombstones.

cleanup_promoted!(redis_key, shard_data_path, keydir, data_dir, shard_index, instance_ctx \\ nil)

@spec cleanup_promoted!(
  binary(),
  binary(),
  atom(),
  binary(),
  non_neg_integer(),
  term()
) :: :ok

dedicated_path(data_dir, shard_index, type, redis_key)

@spec dedicated_path(binary(), non_neg_integer(), atom(), binary()) :: binary()

find_active(path)

@spec find_active(binary()) :: binary()

Returns the active (highest file_id) log file path in a dedicated directory.

marker_key(redis_key)

@spec marker_key(binary()) :: binary()

open_dedicated(data_dir, shard_index, type, redis_key)

@spec open_dedicated(binary(), non_neg_integer(), atom(), binary()) ::
  {:ok, binary()} | {:error, term()}

promote_collection!(type, redis_key, shard_data_path, keydir, data_dir, shard_index, instance_ctx \\ nil)

@spec promote_collection!(
  atom(),
  binary(),
  binary(),
  atom(),
  binary(),
  non_neg_integer(),
  term()
) :: {:ok, reference()} | {:error, term()}

promote_hash!(redis_key, shared_store, keydir, data_dir, shard_index, instance_ctx \\ nil)

@spec promote_hash!(
  binary(),
  reference(),
  atom(),
  binary(),
  non_neg_integer(),
  term()
) ::
  {:ok, reference()} | {:error, term()}

recover_promoted(shard_data_path, keydir, data_dir, shard_index, instance_ctx \\ nil)

@spec recover_promoted(binary(), atom(), binary(), non_neg_integer(), term()) :: map()

threshold()

@spec threshold() :: non_neg_integer()

threshold(ctx)

@spec threshold(FerricStore.Instance.t()) :: non_neg_integer()

Returns the promotion threshold from instance ctx.

with_compaction_latch(owner, redis_key, fun)

@spec with_compaction_latch(map(), binary(), (-> term())) :: term()

Runs fun while holding the per-promoted-key compaction latch.

Promoted dedicated compaction rewrites member records into a newer log file. Raft-applied promoted writes must not append to that same per-key log while the rewrite snapshot is in progress, or disk replay order can resurrect stale compacted records. The latch is only consulted on promoted dedicated paths.