Bedrock.DataPlane.ShardRouter (bedrock v0.7.0)
View SourceRoutes keys to shards and shards to logs using ceiling search and golden ratio distribution.
Shard Lookup
Operates on an immutable :gb_trees shard map of end_key => {tag, start_key}
entries, where end_key is the shard's exclusive upper bound (see
Bedrock.DataPlane.CommitProxy.RoutingData). Because the map is a plain
immutable value, lookups always see one self-consistent snapshot - there is
no shared table for concurrent writers to race on.
To find the shard for a key:
- Find the first entry where
end_key > key - Return that entry's tag
Log Selection
Uses the golden ratio algorithm for deterministic, well-distributed log selection. Given a shard tag, number of logs, and replication factor, returns the indices of logs that should store data for that shard.
The mapping is deterministic and stable as long as the log list order is preserved.
Summary
Functions
First entry with end_key strictly greater than key (end keys are
exclusive bounds), or :none. The one ceiling-walk authority: mutation
routing, key lookup, and the proxy's client-facing covering entry all
resolve through it.
Returns m log indices for shard tag x given n total logs.
The replica set for a shard tag: log ids resolved through log_map
(index → log id over the epoch's sorted log ids — see log_map/1)
via the golden-ratio walk. Non-integer tags are normalized by hashing,
so both consumers agree on the walk's input.
The index map for an epoch's logs: index → log id over the sorted ids.
Looks up the shard tag for a key using ceiling search.
Returns shards overlapping a key range, with their boundaries.
Normalizes a shard tag for the golden-ratio walk. Production tags are integers; anything else (test fixtures) hashes to one so the walk stays defined — and every consumer normalizes identically.
Functions
@spec ceiling_entry(Bedrock.DataPlane.CommitProxy.RoutingData.shard_tree(), binary()) :: {Bedrock.key(), {tag :: term(), start_key :: Bedrock.key()}} | :none
First entry with end_key strictly greater than key (end keys are
exclusive bounds), or :none. The one ceiling-walk authority: mutation
routing, key lookup, and the proxy's client-facing covering entry all
resolve through it.
@spec get_log_indices(non_neg_integer(), pos_integer(), non_neg_integer()) :: [ non_neg_integer() ]
Returns m log indices for shard tag x given n total logs.
Uses golden ratio stepping for uniform distribution. The result is deterministic for the same inputs.
Parameters
x- Shard tag (non-negative integer)n- Total number of logsm- Replication factor (how many logs to return)
Examples
iex> ShardRouter.get_log_indices(0, 5, 2)
[4, 0] # Two distinct indices in range [0, 5)
iex> ShardRouter.get_log_indices(0, 5, 0)
[]
@spec log_ids_for_tag( term(), %{required(non_neg_integer()) => Bedrock.DataPlane.Log.id()}, non_neg_integer() ) :: [Bedrock.DataPlane.Log.id()]
The replica set for a shard tag: log ids resolved through log_map
(index → log id over the epoch's sorted log ids — see log_map/1)
via the golden-ratio walk. Non-integer tags are normalized by hashing,
so both consumers agree on the walk's input.
This is the single site for shard→log placement. Commit-proxy mutation routing and the director's materializer pull-source seeds both resolve through it, so they cannot disagree.
@spec log_map([Bedrock.DataPlane.Log.id()]) :: %{ required(non_neg_integer()) => Bedrock.DataPlane.Log.id() }
The index map for an epoch's logs: index → log id over the sorted ids.
Both consumers of log_ids_for_tag/3 build their map here, so the
sort and indexing cannot diverge between them.
@spec lookup_shard(Bedrock.DataPlane.CommitProxy.RoutingData.shard_tree(), binary()) :: non_neg_integer()
Looks up the shard tag for a key using ceiling search.
Shard ranges are [min, max) - start inclusive, end exclusive.
Parameters
shards- shard map (end_key => {tag, start_key}gb_tree)key- The key to look up
Returns
The shard tag (non-negative integer) that owns the key.
Examples
# Shards: 0 covers ["", "m"), 1 covers ["m", "\xff")
iex> lookup_shard(shards, "apple")
0
iex> lookup_shard(shards, "m")
1 # "m" is the START of shard 1, not end of shard 0
iex> lookup_shard(shards, "zebra")
1
@spec lookup_shards_with_ranges( Bedrock.DataPlane.CommitProxy.RoutingData.shard_tree(), binary(), binary() ) :: [{non_neg_integer(), binary(), binary()}]
Returns shards overlapping a key range, with their boundaries.
Returns {tag, shard_start, shard_end} tuples to enable clamping range
mutations to shard boundaries.
Parameters
shards- shard map (end_key => {tag, start_key}gb_tree)start_key- Start of the range (inclusive)end_key- End of the range (exclusive)
Returns
List of {tag, shard_start, shard_end} tuples for all shards that
overlap [start_key, end_key). Returns [] when the range lies entirely
beyond shard coverage (start_key >= every shard's exclusive end_key).
Examples
# Shards: 0 = ["", "d"), 1 = ["d", "h"), 2 = ["h", "m"), 3 = ["m", "\xff")
iex> lookup_shards_with_ranges(shards, "a", "c")
[{0, "", "d"}]
iex> lookup_shards_with_ranges(shards, "c", "j")
[{0, "", "d"}, {1, "d", "h"}, {2, "h", "m"}]
Normalizes a shard tag for the golden-ratio walk. Production tags are integers; anything else (test fixtures) hashes to one so the walk stays defined — and every consumer normalizes identically.