Bedrock.ObjectStorage.Chunk (bedrock v0.6.0)

Chunk format for storing transaction data in object storage.

Binary Format

Chunks store a sequence of transactions with a header and directory for efficient seeking. The format supports HTTP Range requests by placing the directory before the data.

[Header - 32 bytes]
  - 4 bytes: Magic number (0x42444348 = "BDCH")
  - 1 byte:  Format version (0x01)
  - 1 byte:  Flags (reserved)
  - 2 bytes: Reserved
  - 8 bytes: Min version (lowest txn version in chunk)
  - 8 bytes: Max version (highest txn version in chunk)
  - 4 bytes: Transaction count
  - 4 bytes: Directory size in bytes

[Directory - variable size]
  Array of directory entries, one per transaction:
  - 8 bytes: Transaction version
  - 4 bytes: Data offset from start of data section
  - 4 bytes: Data length

[Data section - variable size]
  Concatenated transaction data (BRDT format)

Read Workflow

Range request for header (32 bytes) to get directory size
Range request for directory (header.directory_size bytes)
Binary search directory for target version
Range request for data from offset to end (or specific range)

Version Ordering

Transactions within a chunk are stored in ascending version order. The chunk is named by its max version (highest version it contains).

Summary

Types

directory_entry()

header()

t()

transaction_data()

version()

Functions

byte_range_from_version(map, target_version)

Calculates the byte range needed to read from a specific version to the end.

decode(binary)

Decodes a complete chunk from binary data.

decode_directory(binary, count)

Decodes directory entries from binary data.

decode_header(arg1)

Decodes a chunk header from binary data.

directory_entry_size()

Returns the size of each directory entry in bytes.

encode(transactions)

Encodes a list of transactions into chunk format.

extract_transactions(map)

Extracts transactions from a decoded chunk.

find_entry(directory, target_version)

Finds the directory entry for a specific version using binary search.

find_first_entry_gte(directory, target_version)

Finds the first directory entry with version >= target.

header_size()

Returns the fixed header size in bytes.

Types

directory_entry()

@type directory_entry() :: %{
  version: version(),
  offset: non_neg_integer(),
  length: non_neg_integer()
}

t()

@type t() :: %{header: header(), directory: [directory_entry()], data: binary()}

transaction_data()

@type transaction_data() :: binary()

version()

@type version() :: non_neg_integer()

Functions

byte_range_from_version(map, target_version)

@spec byte_range_from_version(t(), version()) ::
  {:ok, {non_neg_integer(), non_neg_integer()}} | {:error, :version_not_found}

Calculates the byte range needed to read from a specific version to the end.

Returns {start_byte, end_byte} for use with HTTP Range requests. The range is inclusive on both ends.

decode(binary)

@spec decode(binary()) :: {:ok, t()} | {:error, term()}

Decodes a complete chunk from binary data.

Returns

{:ok, chunk} - Parsed chunk with header, directory, and data
{:error, reason} - Invalid chunk

decode_directory(binary, count)

@spec decode_directory(binary(), non_neg_integer()) ::
  {:ok, [directory_entry()]} | {:error, term()}

Decodes directory entries from binary data.

Parameters

binary - Directory section binary
count - Number of entries to decode

Returns

{:ok, entries} - List of directory entries
{:error, reason} - Invalid directory

decode_header(arg1)

@spec decode_header(binary()) :: {:ok, header()} | {:error, term()}

Decodes a chunk header from binary data.

Returns

{:ok, header} - Parsed header
{:error, reason} - Invalid header

directory_entry_size()

@spec directory_entry_size() :: pos_integer()

Returns the size of each directory entry in bytes.

encode(transactions)

@spec encode([{version(), transaction_data()}]) :: {:ok, binary()} | {:error, term()}

Encodes a list of transactions into chunk format.

Transactions must be provided as {version, data} tuples in ascending version order.

Parameters

transactions - List of {version, binary_data} tuples, sorted by version

Returns

{:ok, chunk_binary} - Encoded chunk
{:error, reason} - Encoding failed

extract_transactions(map)

@spec extract_transactions(t()) :: [{version(), transaction_data()}]

Extracts transactions from a decoded chunk.

Returns a list of {version, data} tuples.

find_entry(directory, target_version)

@spec find_entry([directory_entry()], version()) :: directory_entry() | nil

Finds the directory entry for a specific version using binary search.

Returns the entry if found, or nil if not present.

find_first_entry_gte(directory, target_version)

@spec find_first_entry_gte([directory_entry()], version()) ::
  {non_neg_integer(), directory_entry()} | nil

Finds the first directory entry with version >= target.

Useful for seeking to a position for forward scanning.

header_size()

@spec header_size() :: pos_integer()

Returns the fixed header size in bytes.