raft_kv v0.1.1 RaftKV View Source
An Elixir library to store key-value pairs in a distributed, fault-tolerant, self-adjusting data structure.
Feature & Design
Each value can be arbitrary data structure.
- Operation on a stored value must be given as an implementation of
RaftKV.ValuePerKey
behaviour.
- Operation on a stored value must be given as an implementation of
Built on top of raft_fleet.
- Key-value pairs are sharded into multiple Raft consensus groups by hash partitioning.
- Based on number of keys, data size and current load, shards are automatically split/merged in a transparent manner.
- Designed for many key-value pairs and throughput.
Usage
Suppose we have the following callback module for simple key-value store.
defmodule KV do
alias RaftKV.ValuePerKey
@behaviour ValuePerKey
@impl true
def command(_previous_value, _size, _key, {:set, value}) do
{:ok, 5, value, 0}
end
def command(_previous_value, _size, _key, :unset) do
{:ok, 5, nil, 0}
end
@impl true
def query(value, _size, _key, :get) do
{value, 1}
end
#
# API
#
def get(k) do
case RaftKV.query(:kv, k, :get) do
{:ok, v} -> v
{:error, :key_not_found} -> nil
end
end
def set(k, v) do
{:ok, :ok} = RaftKV.command(:kv, k, {:set, v})
:ok
end
def unset(k) do
{:ok, :ok} = RaftKV.command(:kv, k, :unset)
:ok
end
end
Let’s initialize :raft_kv
and then register a keyspace.
RaftFleet.activate("some_zone")
RaftKV.init()
RaftKV.register_keyspace(:kv, [], KV, nil, %RaftKV.SplitMergePolicy{max_shards: 16, max_keys_per_shard: 100})
Now we can get/set values with arbitrary keys.
KV.get("foo") # => nil
KV.set("foo", "bar") # => :ok
KV.get("foo") # => "bar"
Link to this section Summary
Types
Options for command/4
, query/4
and command_on_all_keys_in_shard/3
Functions
Executes a command on the replicated value identified by keyspace_name
and key
(shard-aware API) Executes a command on all existing keys in the specified shard
Removes an existing keyspace
Retrieves the current value of RaftKV.SplitMergePolicy.t/0
used for the specified keyspace
Initializes :raft_kv
so that processes in Node.self()
can interact with appropriate shard(s) for command/query
(shard-aware API) Fetches all keys in the specified shard
List all registered keyspace names
Executes a read-only query on the replicated value identified by keyspace_name
and key
(shard-aware API) Traverses all shards in the specified keyspace
Registers a keyspace with the given arguments
Replaces the current RaftKV.SplitMergePolicy.t/0
of a keyspace with the specified one
Link to this section Types
option() :: {:timeout, pos_integer()} | {:retry, non_neg_integer()} | {:retry_interval, pos_integer()} | {:call_module, module()} | {:range_shift_retry, non_neg_integer()} | {:range_shift_retry_interval, pos_integer()}
Options for command/4
, query/4
and command_on_all_keys_in_shard/3
.
:timeout
,:retry
,:retry_interval
,:call_module
are directly passed toRaftFleet.command/5
orRaftFleet.query/5
.:shard_lock_retry
and:shard_lock_retry_interval
are intended to mask temporary unavailability of a shard due to an ongoing splitting/merging.
Default values:
:timeout
:500
:retry
:3
:retry_interval
:1000
:call_module
::gen_statem
:shard_lock_retry
:3
:shard_lock_retry_interval
:200
Link to this section Functions
command(atom(), RaftKV.ValuePerKey.key(), RaftKV.ValuePerKey.command_arg(), [ option() ]) :: {:ok, RaftKV.ValuePerKey.command_ret()} | {:error, :no_leader}
Executes a command on the replicated value identified by keyspace_name
and key
.
See also RaftKV.ValuePerKey
, RaftFleet.command/6
and RaftedValue.command/5
.
command_on_all_keys_in_shard(atom(), RaftKV.ValuePerKey.command_arg(), [ option() ]) :: :ok | {:error, :no_leader}
(shard-aware API) Executes a command on all existing keys in the specified shard.
Note that values of RaftKV.ValuePerKey.command_ret
that were returned by existing keys’ command
are not returned to the caller of this function.
deregister_keyspace(atom()) :: :ok | {:error, :no_such_keyspace}
Removes an existing keyspace.
Associated resources (processes, ETS records, etc.) for the keyspace are not removed immediately; they are removed by a background worker process.
get_keyspace_policy(atom()) :: nil | RaftKV.SplitMergePolicy.t()
Retrieves the current value of RaftKV.SplitMergePolicy.t/0
used for the specified keyspace.
Initializes :raft_kv
so that processes in Node.self()
can interact with appropriate shard(s) for command/query.
:raft_kv
heavily depends on :raft_fleet
and thus it’s necessary to call RaftFleet.activate/1
before calling this function.
list_keys_in_shard(atom()) :: [RaftKV.ValuePerKey.key()]
(shard-aware API) Fetches all keys in the specified shard.
List all registered keyspace names.
query(atom(), RaftKV.ValuePerKey.key(), RaftKV.ValuePerKey.query_arg(), [ option() ]) :: {:ok, RaftKV.ValuePerKey.query_ret()} | {:error, :key_not_found | :no_leader}
Executes a read-only query on the replicated value identified by keyspace_name
and key
.
See also RaftKV.ValuePerKey
, RaftFleet.query/6
and RaftedValue.query/4
.
(shard-aware API) Traverses all shards in the specified keyspace.
register_keyspace( atom(), Keyword.t(), module(), nil | module(), RaftKV.SplitMergePolicy.t() ) :: :ok | {:error, :invalid_policy | :already_registered}
Registers a keyspace with the given arguments.
Parameters:
keyspace_name
: An atom that identifies the new keyspace.rv_config_options
: A keyword list of options passed toRaftedValue.make_config/2
. The given options are used by all consensus groups in the newly-registered keyspace.data_module
: A callback module that implementsRaftKV.ValuePerKey
behaviour.hook_module
: A callback module that implementsRaftKV.LeaderHook
behaviour (ornil
if you don’t use hook).policy
:RaftKV.SplitMergePolicy.t/0
that specifies the conditions on which shards are split/merged. See alsoRaftKV.SplitMergePolicy
.
set_keyspace_policy(atom(), RaftKV.SplitMergePolicy.t()) :: :ok | {:error, :invalid_policy | :no_such_keyspace}
Replaces the current RaftKV.SplitMergePolicy.t/0
of a keyspace with the specified one.