Ferricstore.Store.Promotion (ferricstore v0.3.2)

Copy Markdown View Source

Collection promotion: migrates large compound-key collections from the shared shard Bitcask into a dedicated per-key Bitcask instance.

Background (spec section 2B.4b)

Small collections (hashes, sets, sorted sets) are stored as compound keys in the shared shard Bitcask (H:key\0field, S:key\0member, Z:key\0member). When any compound-key collection exceeds the configurable promotion threshold (default: 100 entries), it is promoted to a dedicated Bitcask instance stored under:

dedicated/shard_N/{type}:{sha256_of_key}/

where {type} is hash, set, or zset.

Promotion is one-way -- once promoted, a collection stays in its dedicated instance even if entries are later deleted below the threshold. The dedicated instance is only removed when the entire key is deleted via DEL / UNLINK.

Lists are not promoted

Lists store all elements as a single serialized Erlang term in one Bitcask entry (via ListOps). Since there is no compound key fan-out, a list with 1000 elements is still a single Bitcask entry and does not benefit from promotion. List promotion is intentionally skipped.

Promotion marker

When a key is promoted, a marker entry PM:redis_key is written to the shared Bitcask with the type as its value ("hash", "set", or "zset"). This allows the shard to rediscover promoted keys on restart by scanning for PM: prefixed keys during initialization.

Configuration

config :ferricstore, :promotion_threshold, 100

Set to 0 to disable automatic promotion entirely (no collections will ever be promoted).

Summary

Functions

cleanup_promoted!(redis_key, shard_data_path, keydir, data_dir, shard_index)
dedicated_path(data_dir, shard_index, type, redis_key)
find_active(path)

Returns the active (highest file_id) log file path in a dedicated directory.

marker_key(redis_key)
open_dedicated(data_dir, shard_index, type, redis_key)
promote_collection!(type, redis_key, shard_data_path, keydir, data_dir, shard_index)
promote_hash!(redis_key, shared_store, keydir, data_dir, shard_index)
recover_promoted(shard_data_path, keydir, data_dir, shard_index)
threshold()
threshold(ctx)

Returns the promotion threshold from instance ctx.

Functions

cleanup_promoted!(redis_key, shard_data_path, keydir, data_dir, shard_index)

@spec cleanup_promoted!(binary(), binary(), atom(), binary(), non_neg_integer()) ::
  :ok

dedicated_path(data_dir, shard_index, type, redis_key)

@spec dedicated_path(binary(), non_neg_integer(), atom(), binary()) :: binary()

find_active(path)

@spec find_active(binary()) :: binary()

Returns the active (highest file_id) log file path in a dedicated directory.

marker_key(redis_key)

@spec marker_key(binary()) :: binary()

open_dedicated(data_dir, shard_index, type, redis_key)

@spec open_dedicated(binary(), non_neg_integer(), atom(), binary()) ::
  {:ok, binary()} | {:error, term()}

promote_collection!(type, redis_key, shard_data_path, keydir, data_dir, shard_index)

@spec promote_collection!(
  atom(),
  binary(),
  binary(),
  atom(),
  binary(),
  non_neg_integer()
) ::
  {:ok, reference()} | {:error, term()}

promote_hash!(redis_key, shared_store, keydir, data_dir, shard_index)

@spec promote_hash!(binary(), reference(), atom(), binary(), non_neg_integer()) ::
  {:ok, reference()} | {:error, term()}

recover_promoted(shard_data_path, keydir, data_dir, shard_index)

@spec recover_promoted(binary(), atom(), binary(), non_neg_integer()) :: map()

threshold()

@spec threshold() :: non_neg_integer()

threshold(ctx)

@spec threshold(FerricStore.Instance.t()) :: non_neg_integer()

Returns the promotion threshold from instance ctx.