Bedrock.ObjectStorage.SnapshotBundle (bedrock v0.5.3)
View SourceUtilities for working with snapshot bundles.
A snapshot bundle is a single file containing: [data bytes] [index record] <- IndexDatabase format (magic 0x4F4C5644)
This module provides utilities to split a downloaded bundle into separate data and index files for use with the local Olivine storage engine.
Summary
Functions
Create a bundle file from separate data and index files.
Find where the index record starts in a bundle.
Find the index boundary by reading only the footer and header from a file.
Split a bundle file into separate data and index files.
Split a bundle file in place, avoiding data copies.
Functions
@spec create(data_path :: Path.t(), idx_path :: Path.t(), bundle_path :: Path.t()) :: {:ok, bundle_size :: non_neg_integer()} | {:error, term()}
Create a bundle file from separate data and index files.
Concatenates data and index into a single bundle file. Returns {:ok, bundle_size} on success.
@spec find_index_boundary(binary()) :: {:ok, data_end_offset :: non_neg_integer(), idx_size :: non_neg_integer()} | {:error, :invalid_bundle | :no_index_record}
Find where the index record starts in a bundle.
Returns {:ok, data_end_offset, idx_size} where:
- data_end_offset: Byte offset where data ends and index begins
- idx_size: Size of the index record in bytes
@spec find_index_boundary_from_file(Path.t(), non_neg_integer()) :: {:ok, data_end_offset :: non_neg_integer(), idx_size :: non_neg_integer()} | {:error, :invalid_bundle | :no_index_record | term()}
Find the index boundary by reading only the footer and header from a file.
More efficient than find_index_boundary/1 for large files since it
doesn't read the entire bundle into memory.
@spec split(bundle_path :: Path.t(), data_path :: Path.t(), idx_path :: Path.t()) :: {:ok, data_size :: non_neg_integer(), idx_size :: non_neg_integer()} | {:error, term()}
Split a bundle file into separate data and index files.
Reads the bundle, identifies the index record at the end, and writes:
- data_path: All bytes before the index record
- idx_path: The complete index record
Returns {:ok, data_size, idx_size} on success.
@spec split_in_place( bundle_path :: Path.t(), data_path :: Path.t(), idx_path :: Path.t() ) :: {:ok, data_size :: non_neg_integer(), idx_size :: non_neg_integer()} | {:error, term()}
Split a bundle file in place, avoiding data copies.
This is more efficient than split/3 for large bundles:
- Reads only the index portion (small) from the end of the bundle
- Writes index to a separate file
- Truncates the bundle at the data boundary (no copy!)
- Renames the truncated bundle to the data path
The bundle file is consumed (renamed to data_path). Returns {:ok, data_size, idx_size} on success.