Bedrock.DataPlane.Transaction (bedrock v0.5.2)
View SourceTagged binary transaction encoding for Bedrock that supports efficient operations and extensibility through self-describing sections with embedded CRC validation.
This module implements a comprehensive tagged binary format that encodes the
simple map structure from Tx.commit/1 with an efficient binary format featuring:
- Tagged sections: Self-describing sections with type, size, and embedded CRC
- Order independence: Sections can appear in any order for better extensibility
- Section omission: All sections are optional - empty sections are completely omitted to save space
- Efficient operations: Extract specific sections without full decode
- Robust validation: Self-validating CRCs detect any bit corruption
- 16-bit instruction format: 5-bit operations + 3-bit reserved + 8-bit parameters with optimized length encoding
Transaction Structure
Input/output transaction map:
%{
commit_version: Bedrock.version() # assigned by commit proxy
mutations: [Tx.mutation()], # {:set, binary(), binary()} | {:clear_range, binary(), binary()} | atomic operations
read_conflicts: {read_version, [key_range()]},
write_conflicts: [key_range()}],
}Binary Format
[OVERALL HEADER - 8 bytes]
- 4 bytes: Magic number (0x42524454 = "BRDT")
- 1 byte: Format version (0x01)
- 1 byte: Flags (reserved, set to 0)
- 2 bytes: Section count (big-endian)
[SECTION 1: header + payload]
[SECTION 2: header + payload]
...
[SECTION N: header + payload]Section Format
Each section has an 8-byte header with CRC validation:
[SECTION HEADER - 8 bytes]
- 1 byte: Section tag
- 3 bytes: Section payload size (24-bit big-endian, max 16MB)
- 4 bytes: Section CRC32 (standard CRC32 over tag + size + payload)
[SECTION PAYLOAD - variable size]
- Section-specific data formatCRC Validation
Uses standard CRC32 calculation and validation:
- Encoding: Calculate CRC32 over tag + size + payload, store value directly
- Validation: Recalculate CRC32 and compare with stored value
- Coverage: CRC covers tag, size, and payload for robust error detection
Section Types
- 0x01: MUTATIONS - Nearly always present (may be omitted if empty)
- 0x02: READ_CONFLICTS - Present when read conflicts exist AND read version available
- 0x03: WRITE_CONFLICTS - Present when write conflicts exist
- 0x04: COMMIT_VERSION - Present when stamped by commit proxy
Section Usage by Component
Logs: mutations, commit_version Resolver: write_conflicts, read_conflicts (if any), commit_version Commit Proxy: mutations, read_conflicts (if reads performed), write_conflicts
16-bit Header Mutation Format
All mutations use a 16-bit header format with 4-tier variable-length encoding:
Header Structure (16 bits):
- 5 bits: opcode (0-31 operations)
- 3 bits: reserved (future extensibility)
- 8 bits: parameters (format encoding - see below)
Operation Codes:
- 0x00: SET - Set key to value
- 0x01: CLEAR - Clear single key
- 0x02: CLEAR_RANGE - Clear key range
- 0x03: ATOMIC_ADD - Atomic add operation
- 0x04: ATOMIC_MIN - Atomic minimum operation
- 0x05: ATOMIC_MAX - Atomic maximum operation
Parameters Field Encoding:
For two-parameter mutations (SET, CLEAR_RANGE):
- Upper 4 bits (f1::4): Format for first parameter (key/start_key)
- Lower 4 bits (f2::4): Format for second parameter (value/end_key)
For single-parameter mutations (CLEAR, atomics):
- Upper 4 bits (f::4): Format for parameter (key)
- Lower 4 bits: Zero (reserved)
4-Tier Variable-Length Format Values:
- Direct encoding (0-11): Values 0-11 encoded directly as format value
- 1-byte extended (1-256): Format 12 (0b1100) + (len-1) in next byte
- 1-byte extended (257-512): Format 13 (0b1101) + (len-1 & 0xFF) in next byte, 9th bit encoded in format
- 2-byte extended (1-65536): Format 14 (0b1110) + (len-1) in next 2 bytes
- 2-byte extended (65537-131072): Format 15 (0b1111) + (len-1 & 0xFFFF) in next 2 bytes, 17th bit encoded in format
The system automatically selects the most compact format for each parameter.
Summary
Functions
Adds a commit version to an existing transaction.
Adds a new section to the transaction.
Extracts the commit version if present.
Extracts the commit version from an encoded transaction, raising on error.
Decodes a tagged binary transaction back to the transaction map format.
Returns a minimal header-only transaction with no sections for version advancement.
Encodes a transaction map into the tagged binary format.
Extracts a specific section payload by tag without full transaction decode.
Efficiently extracts specific sections from a transaction into a smaller binary transaction.
Efficiently extracts specific sections from a transaction into a smaller binary transaction.
Checks if a mutation affects metadata keys (keys with \xFF prefix).
Creates a stream of mutations from the MUTATIONS section.
Streams mutations from the transaction, raising if the transaction is invalid.
Extracts read conflicts and read version from READ_CONFLICTS section.
Extracts both read and write conflicts in a single pass for optimal performance.
Efficiently extracts specific sections from a transaction and reassembles them into a new transaction.
Extracts the shard index from the SHARD_INDEX section.
Extracts the shard index from an encoded transaction, raising on error.
Validates the binary format integrity using section CRCs.
Extracts write conflicts from WRITE_CONFLICTS section.
Types
@type encoded() :: binary()
@type section_tag() :: 1..255
@type transaction_map() :: Bedrock.transaction_map()
Functions
Adds a commit version to an existing transaction.
@spec add_section(binary(), section_tag(), binary()) :: {:ok, binary()} | {:error, reason :: term()}
Adds a new section to the transaction.
Returns error if section already exists. Use for adding COMMIT_VERSION section after commit proxy processing.
Extracts the commit version if present.
Returns nil if no COMMIT_VERSION section exists.
Extracts the commit version from an encoded transaction, raising on error.
This is the bang version of commit_version/1 that raises an exception
instead of returning an error tuple.
Returns the commit version binary or nil if no commit version is present. Raises an exception if the transaction is malformed.
@spec decode(binary()) :: {:ok, transaction_map()} | {:error, reason :: term()}
Decodes a tagged binary transaction back to the transaction map format.
Validates all section CRCs and handles missing sections appropriately.
@spec empty_transaction() :: encoded()
Returns a minimal header-only transaction with no sections for version advancement.
This is more efficient than encoding an empty transaction map as it contains only the transaction header with zero sections, making it the smallest possible valid transaction binary.
@spec encode(transaction_map()) :: binary()
Encodes a transaction map into the tagged binary format.
Automatically selects the most compact opcode variants based on data sizes and omits empty sections for optimal space efficiency.
Options
:include_commit_version- Include placeholder commit version section:include_transaction_id- DEPRECATED: Use:include_commit_versioninstead
@spec extract_section(binary(), section_tag()) :: {:ok, binary()} | {:error, reason :: term()}
Extracts a specific section payload by tag without full transaction decode.
Efficiently extracts specific sections from a transaction into a smaller binary transaction.
This is a convenience wrapper around reassemble_sections/3 for the common case of extracting sections without adding new ones.
Parameters
encoded_transaction: The source transactionsections_to_keep: List of section names to keep (:mutations,:read_conflicts,:write_conflicts,:commit_version)
Examples
# Extract just conflict sections for resolver
{:ok, conflict_binary} = extract_sections(transaction, [:read_conflicts, :write_conflicts])
# Extract mutations for log processing
{:ok, mutations_binary} = extract_sections(transaction, [:mutations])
Efficiently extracts specific sections from a transaction into a smaller binary transaction.
Same as extract_sections/2 but raises on error instead of returning {:error, reason}.
Parameters
encoded_transaction: The source transactionsections_to_keep: List of section names to keep (:mutations,:read_conflicts,:write_conflicts,:commit_version)
Examples
# Extract just conflict sections for resolver
conflict_binary = extract_sections!(transaction, [:read_conflicts, :write_conflicts])
# Extract mutations for log processing
mutations_binary = extract_sections!(transaction, [:mutations])
@spec metadata_mutation?(Bedrock.Internal.TransactionBuilder.Tx.mutation()) :: boolean()
Checks if a mutation affects metadata keys (keys with \xFF prefix).
Returns true if any key in the mutation starts with \xFF.
For :clear_range, returns true if either the start or end key has the prefix.
Examples
iex> Transaction.metadata_mutation?({:set, <<0xFF, "foo">>, "value"})
true
iex> Transaction.metadata_mutation?({:set, "user_key", "value"})
false
iex> Transaction.metadata_mutation?({:clear_range, "start", <<0xFF, "end">>})
true
@spec mutations(binary()) :: {:ok, Enumerable.t(Bedrock.Internal.TransactionBuilder.Tx.mutation())} | {:error, reason :: term()}
Creates a stream of mutations from the MUTATIONS section.
Enables processing large transactions without loading all mutations into memory.
@spec mutations!(binary()) :: Enumerable.t(Bedrock.Internal.TransactionBuilder.Tx.mutation())
Streams mutations from the transaction, raising if the transaction is invalid.
Returns a stream of mutations. Use this when you're confident the transaction is valid or want to fail fast on invalid data.
@spec read_conflicts(binary()) :: {:ok, {Bedrock.version() | nil, [{binary(), binary()}]}} | {:error, reason :: term()}
Extracts read conflicts and read version from READ_CONFLICTS section.
Returns {nil, []} if no READ_CONFLICTS section exists.
@spec read_write_conflicts(binary()) :: {:ok, {{Bedrock.version() | nil, [Bedrock.key_range()]}, [Bedrock.key_range()]}} | {:error, reason :: term()}
Extracts both read and write conflicts in a single pass for optimal performance.
This is more efficient than calling read_conflicts/1 and write_conflicts/1 separately as it only parses the transaction once.
Returns a tuple with read info and write conflicts.
@spec reassemble_sections(binary(), [atom()], %{required(atom()) => binary()}) :: {:ok, binary()} | {:error, reason :: term()}
Efficiently extracts specific sections from a transaction and reassembles them into a new transaction.
This avoids the decode-encode cycle by working directly with binary sections. Much faster than extracting data and calling encode/1 again.
Parameters
encoded_transaction: The source transactionsections_to_keep: List of section names to keep in the new transaction (:mutations,:read_conflicts,:write_conflicts,:commit_version)new_sections: Map of additional sections to add (e.g. %{commit_version: version_binary})
Examples
# Extract just mutations and add commit version
{:ok, log_transaction} = reassemble_sections(transaction, [:mutations], %{commit_version: version})
# Extract read conflicts and write conflicts for resolver
{:ok, conflict_data} = reassemble_sections(transaction, [:read_conflicts, :write_conflicts], %{})
Extracts the shard index from the SHARD_INDEX section.
Returns a list of {tag, count} tuples indicating how many contiguous
mutations belong to each shard. Returns nil if no SHARD_INDEX section exists.
Example
{:ok, [{0, 5}, {1, 3}]} = Transaction.shard_index(encoded_tx)
# First 5 mutations are shard 0, next 3 are shard 1
Extracts the shard index from an encoded transaction, raising on error.
Validates the binary format integrity using section CRCs.
Each section validates independently using standard CRC validation by recalculating CRC32 over tag + size + payload and comparing with stored CRC.
Extracts write conflicts from WRITE_CONFLICTS section.
Returns empty list if no WRITE_CONFLICTS section exists.