Bedrock.ObjectStorage.SnapshotBundle (bedrock v0.5.3)

View Source

Utilities for working with snapshot bundles.

A snapshot bundle is a single file containing: [data bytes] [index record] <- IndexDatabase format (magic 0x4F4C5644)

This module provides utilities to split a downloaded bundle into separate data and index files for use with the local Olivine storage engine.

Summary

Functions

Create a bundle file from separate data and index files.

Find where the index record starts in a bundle.

Find the index boundary by reading only the footer and header from a file.

Split a bundle file into separate data and index files.

Split a bundle file in place, avoiding data copies.

Functions

create(data_path, idx_path, bundle_path)

@spec create(data_path :: Path.t(), idx_path :: Path.t(), bundle_path :: Path.t()) ::
  {:ok, bundle_size :: non_neg_integer()} | {:error, term()}

Create a bundle file from separate data and index files.

Concatenates data and index into a single bundle file. Returns {:ok, bundle_size} on success.

find_index_boundary(bundle)

@spec find_index_boundary(binary()) ::
  {:ok, data_end_offset :: non_neg_integer(), idx_size :: non_neg_integer()}
  | {:error, :invalid_bundle | :no_index_record}

Find where the index record starts in a bundle.

Returns {:ok, data_end_offset, idx_size} where:

  • data_end_offset: Byte offset where data ends and index begins
  • idx_size: Size of the index record in bytes

find_index_boundary_from_file(path, file_size)

@spec find_index_boundary_from_file(Path.t(), non_neg_integer()) ::
  {:ok, data_end_offset :: non_neg_integer(), idx_size :: non_neg_integer()}
  | {:error, :invalid_bundle | :no_index_record | term()}

Find the index boundary by reading only the footer and header from a file.

More efficient than find_index_boundary/1 for large files since it doesn't read the entire bundle into memory.

split(bundle_path, data_path, idx_path)

@spec split(bundle_path :: Path.t(), data_path :: Path.t(), idx_path :: Path.t()) ::
  {:ok, data_size :: non_neg_integer(), idx_size :: non_neg_integer()}
  | {:error, term()}

Split a bundle file into separate data and index files.

Reads the bundle, identifies the index record at the end, and writes:

  • data_path: All bytes before the index record
  • idx_path: The complete index record

Returns {:ok, data_size, idx_size} on success.

split_in_place(bundle_path, data_path, idx_path)

@spec split_in_place(
  bundle_path :: Path.t(),
  data_path :: Path.t(),
  idx_path :: Path.t()
) ::
  {:ok, data_size :: non_neg_integer(), idx_size :: non_neg_integer()}
  | {:error, term()}

Split a bundle file in place, avoiding data copies.

This is more efficient than split/3 for large bundles:

  1. Reads only the index portion (small) from the end of the bundle
  2. Writes index to a separate file
  3. Truncates the bundle at the data boundary (no copy!)
  4. Renames the truncated bundle to the data path

The bundle file is consumed (renamed to data_path). Returns {:ok, data_size, idx_size} on success.