Bedrock.DataPlane.Materializer.Olivine.IndexManager (bedrock v0.5.2)

View Source

Page management core for the Olivine storage driver.

Implements Phase 2.1 of the Olivine implementation plan:

  • 5-second sliding time window for version retention
  • Version advancement and window expiry
  • Page eviction when versions exit window with efficient page collection
  • Version filtering for queries
  • Binary page encoding/decoding with 32-byte header format
  • Page creation and key lookup within pages
  • Simple median split algorithm (256 key threshold)
  • Page ID allocation with max_id tracking

The output queue stores modified pages alongside version metadata to enable efficient collection during eviction without redundant filtering operations.

Summary

Types

loader_fn()
modified_pages()
operation()
page()
page_id()
t()
version_data()
version_list()
version_update_data()

Functions

advance_window(index_manager, max_eviction_size_bytes)

Advances the window by determining what to evict and updating both buffer tracking and hot set. This is the complete window advancement operation that combines

apply_transaction(index_manager, transaction, database)

Applies a single transaction binary to the version manager. Creates a new version and applies all mutations in the transaction. Uses a two-pass approach: first collect all instructions, then process each page.

apply_transactions(index_manager, transactions, database)
get_complete_page_map(map)

Extracts the complete page_map from the current version's index. Used during compaction to get all current pages.

info(index_manager, stat)
new()
page_for_key(index_manager, key, version)
pages_for_range(index_manager, start_key, end_key, version)
recover_from_database(database)

Types

loader_fn()

@type loader_fn() :: (Bedrock.key(), Bedrock.version() ->
                  {:ok, Bedrock.value()} | {:error, :not_found})

modified_pages()

@type modified_pages() :: %{
  required(Bedrock.DataPlane.Materializer.Olivine.Index.Page.id()) =>
    {Bedrock.DataPlane.Materializer.Olivine.Index.Page.t(),
     Bedrock.DataPlane.Materializer.Olivine.Index.Page.id()}
}

operation()

@type operation() :: {:set, Bedrock.version()} | :clear

page()

@type page() :: Bedrock.DataPlane.Materializer.Olivine.Index.Page.t()

page_id()

@type page_id() :: Bedrock.DataPlane.Materializer.Olivine.Index.Page.id()

t()

@type t() :: %Bedrock.DataPlane.Materializer.Olivine.IndexManager{
  current_version: Bedrock.version(),
  id_allocator: Bedrock.DataPlane.Materializer.Olivine.IdAllocator.t(),
  last_version_ended_at_offset: non_neg_integer(),
  n_keys: non_neg_integer(),
  output_queue: :queue.queue(),
  versions: version_list(),
  window_lag_time_μs: pos_integer(),
  window_size_in_microseconds: pos_integer()
}

version_data()

@type version_data() ::
  {Bedrock.DataPlane.Materializer.Olivine.Index.t(), modified_pages()}

version_list()

@type version_list() :: [{Bedrock.version(), version_data()}]

version_update_data()

@type version_update_data() :: Bedrock.DataPlane.Materializer.Olivine.IndexUpdate.t()

Functions

advance_window(index_manager, max_eviction_size_bytes)

@spec advance_window(t(), pos_integer()) ::
  {:no_eviction, t()}
  | {:evict, non_neg_integer(), t(), [any()], Bedrock.version()}

Advances the window by determining what to evict and updating both buffer tracking and hot set. This is the complete window advancement operation that combines:

  1. Calculating window edge (newest version in buffer - 5 seconds)
  2. Determining eviction batch based on size and time constraints, collecting modified pages
  3. Trimming hot set to match eviction point

Returns either {:no_eviction, updated_manager} or {:evict, evicted_count, updated_manager, collected_pages, eviction_version}. The collected_pages contain all modified pages from evicted versions for efficient persistence.

apply_transaction(index_manager, transaction, database)

@spec apply_transaction(
  t(),
  binary(),
  Bedrock.DataPlane.Materializer.Olivine.Database.t()
) ::
  {t(), Bedrock.DataPlane.Materializer.Olivine.Database.t()}

Applies a single transaction binary to the version manager. Creates a new version and applies all mutations in the transaction. Uses a two-pass approach: first collect all instructions, then process each page.

apply_transactions(index_manager, transactions, database)

@spec apply_transactions(
  index_manager :: t(),
  encoded_transactions :: [binary()],
  database :: Bedrock.DataPlane.Materializer.Olivine.Database.t()
) :: {t(), Bedrock.DataPlane.Materializer.Olivine.Database.t()}

get_complete_page_map(map)

@spec get_complete_page_map(t()) :: %{
  required(Bedrock.DataPlane.Materializer.Olivine.Index.Page.id()) =>
    {Bedrock.DataPlane.Materializer.Olivine.Index.Page.t(),
     Bedrock.DataPlane.Materializer.Olivine.Index.Page.id()}
}

Extracts the complete page_map from the current version's index. Used during compaction to get all current pages.

info(index_manager, stat)

@spec info(index_manager :: t(), atom()) :: term()

new()

@spec new() :: t()

page_for_key(index_manager, key, version)

@spec page_for_key(t(), key :: Bedrock.key(), Bedrock.version()) ::
  {:ok, Bedrock.DataPlane.Materializer.Olivine.Index.Page.t()}
  | {:error, :not_found}
  | {:error, :version_too_new}
@spec page_for_key(t(), Bedrock.KeySelector.t(), Bedrock.version()) ::
  {:ok, resolved_key :: binary(),
   Bedrock.DataPlane.Materializer.Olivine.Index.Page.t()}
  | {:partial, keys_available :: non_neg_integer()}
  | {:error, :not_found | :version_too_new | :version_too_old}

pages_for_range(index_manager, start_key, end_key, version)

@spec pages_for_range(
  t(),
  start_key :: Bedrock.key(),
  end_key :: Bedrock.key(),
  Bedrock.version()
) ::
  {:ok, [Bedrock.DataPlane.Materializer.Olivine.Index.Page.t()]}
  | {:error, :version_too_new}
  | {:error, :version_too_old}
@spec pages_for_range(
  t(),
  Bedrock.KeySelector.t(),
  Bedrock.KeySelector.t(),
  Bedrock.version()
) ::
  {:ok, {resolved_start :: binary(), resolved_end :: binary()},
   [Bedrock.DataPlane.Materializer.Olivine.Index.Page.t()]}
  | {:error, :version_too_new | :version_too_old | :invalid_range}

recover_from_database(database)

@spec recover_from_database(
  database :: Bedrock.DataPlane.Materializer.Olivine.Database.t()
) ::
  {:ok, t()} | {:error, :missing_pages}