Nebulex.Adapters.Partitioned (Nebulex.Distributed v3.0.0-rc.1)
View SourceAdapter module for the partitioned cache topology.
Features
- Partitioned cache topology (Sharding Distribution Model).
ExHashRing
for distributing the keys across the cluster members.- Support for transactions via Erlang global name registration facility.
- Configurable primary storage adapter.
Partitioned Cache Topology
There are several key points to consider about a partitioned cache:
Partitioned: The data in a distributed cache is spread out over all the servers in such a way that no two servers are responsible for the same piece of cached data. This means that the size of the cache and the processing power associated with the management of the cache can grow linearly with the size of the cluster. Also, it means that operations against data in the cache can be accomplished with a "single hop," in other words, involving at most one other server.
Load-Balanced: Since the data is spread out evenly over the servers, the responsibility for managing the data is automatically load-balanced across the cluster.
Ownership: Exactly one node in the cluster is responsible for each piece of data in the cache.
Point-To-Point: The communication for the partitioned cache is all point-to-point, enabling linear scalability.
Location Transparency: Although the data is spread out across cluster nodes, the exact same API is used to access the data, and the same behavior is provided by each of the API methods. This is called location transparency, which means that the developer does not have to code based on the topology of the cache, since the API and its behavior will be the same with a local cache, a replicated cache, or a distributed cache.
Failover: Failover of a distributed cache involves promoting backup data to be primary storage. When a cluster node fails, all remaining cluster nodes determine what data each holds in backup that the failed cluster node had primary responsible for when it died. Those data becomes the responsibility of whatever cluster node was the backup for the data. However, this adapter does not provide fault-tolerance implementation, each piece of data is kept in a single node/machine (via sharding), then, if a node fails, the data kept by this node won't be available for the rest of the cluster members.
Based on "Distributed Caching Essential Lessons" by Cameron Purdy and Coherence Partitioned Cache Service.
Additional implementation notes
:pg
is used under-the-hood by the adapter to manage the cluster nodes.
When the partitioned cache is started in a node, it creates a group and joins
it (the cache supervisor PID is joined to the group). Then, when a function
is invoked, the adapter uses ExHashRing
to determine which node should
handle the request based on the key's hash value. This ensures consistent
key distribution across the cluster nodes, even when nodes join or leave
the cluster.
The key distribution process works as follows:
- Each node in the cluster is assigned a set of virtual nodes (vnodes) in the hash ring.
- When a key is accessed,
ExHashRing.Ring
is used to find the node responsible for that key (the hash value is used to find the corresponding vnode in the hash ring). - The request is routed to the physical node that owns that vnode.
This consistent hashing approach provides several benefits:
- Minimal key redistribution when nodes join or leave the cluster.
- Even distribution of keys across the cluster.
- Predictable key-to-node mapping.
- Efficient node lookup for key operations.
When a partitioned cache supervisor dies (the cache is stopped or killed for some reason), the PID of that process is automatically removed from the PG group. The hash ring is then automatically rebalanced to ensure keys are properly distributed among the remaining nodes.
This adapter depends on a local cache adapter (primary storage), it adds
an extra layer on top of it in order to distribute requests across a group
of nodes, where is supposed the local cache is running already. However,
you don't need to define any additional cache module for the primary
storage, instead, the adapter initializes it automatically (it adds the
primary storage as part of the supervision tree) based on the given
:primary_storage_adapter
option.
Usage
The cache expects the :otp_app
and :adapter
as options when used.
The :otp_app
should point to an OTP application with the cache
configuration. Optionally, you can configure the desired primary
storage adapter with the option :primary_storage_adapter
(defaults to Nebulex.Adapters.Local
). See the compile time options
for more information:
:primary_storage_adapter
(atom/0
) - The adapter for the primary storage. The default value isNebulex.Adapters.Local
.
For example:
defmodule MyApp.PartitionedCache do
use Nebulex.Cache,
otp_app: :my_app,
adapter: Nebulex.Adapters.Partitioned
end
Providing the :primary_storage_adapter
:
defmodule MyApp.PartitionedCache do
use Nebulex.Cache,
otp_app: :my_app,
adapter: Nebulex.Adapters.Partitioned,
primary_storage_adapter: Nebulex.Adapters.Local
end
Where the configuration for the cache must be in your application environment,
usually defined in your config/config.exs
:
config :my_app, MyApp.PartitionedCache,
primary: [
gc_interval: 3_600_000,
backend: :shards
]
If your application was generated with a supervisor (by passing --sup
to mix new
) you will have a lib/my_app/application.ex
file containing
the application start callback that defines and starts your supervisor.
You just need to edit the start/2
function to start the cache as a
supervisor on your application's supervisor:
def start(_type, _args) do
children = [
{MyApp.PartitionedCache, []},
...
]
See Nebulex.Cache
for more information.
Configuration options
This adapter supports the following configuration options:
:primary
(keyword/0
) - Options for the adapter configured via the:primary_storage_adapter
option. The options will vary depending on the adapter used. The default value is[]
.:hash_ring
(keyword/0
) - Options for the hash ring. SeeExHashRing.Ring.start_link/2
for more information.The default value is
[]
.
Shared runtime options
When using the partitioned adapter, all of the cache functions outlined in
Nebulex.Cache
accept the following options:
:timeout
(timeout/0
) - The time in milliseconds to wait for a command to finish (:infinity
to wait indefinitely). The default value is5000
.
Stream options
The stream
command supports the following options:
:on_error
(:raise
|:nothing
) - Indicates whether to raise an exception when an error occurs or do nothing (skip errors).When the stream is evaluated, the adapter attempts to execute the
stream
command on the different nodes. Still, the execution could fail due to an RPC error or the command explicitly returns an error. If the option is set to:raise
, the command will raise an exception when an error occurs on the stream evaluation. On the other hand, if it is set to:nothing
, the error is skipped.The default value is
:raise
.
Telemetry events
Since the partitioned adapter depends on the configured primary storage
cache (which uses a local cache adapter), this one will also emit Telemetry
events. Therefore, there will be events emitted by the partitioned adapter
as well as the primary storage cache. For example, the cache defined before
MyApp.PartitionedCache
will emit the following events:
[:my_app, :partitioned_cache, :command, :start]
[:my_app, :partitioned_cache, :primary, :command, :start]
[:my_app, :partitioned_cache, :command, :stop]
[:my_app, :partitioned_cache, :primary, :command, :stop]
[:my_app, :partitioned_cache, :command, :exception]
[:my_app, :partitioned_cache, :primary, :command, :exception]
As you may notice, the telemetry prefix by default for the cache is
[:my_app, :partitioned_cache]
. However, you could specify the
:telemetry_prefix
for the primary storage within the :primary
options
(if you want to override the default). See the
Telemetry guide
for more information and examples.
Adapter-specific telemetry events
This adapter exposes following Telemetry events:
telemetry_prefix ++ [:bootstrap, :started]
- Dispatched by the adapter when the bootstrap process is started.Measurements:
%{system_time: non_neg_integer}
Metadata:
%{ adapter_meta: %{optional(atom) => term}, cluster_nodes: [node] }
telemetry_prefix ++ [:bootstrap, :stopped]
- Dispatched by the adapter when the bootstrap process is stopped.Measurements:
%{system_time: non_neg_integer}
Metadata:
%{ adapter_meta: %{optional(atom) => term}, cluster_nodes: [node], reason: term }
telemetry_prefix ++ [:bootstrap, :exit]
- Dispatched by the adapter when the bootstrap has received an exit signal.Measurements:
%{system_time: non_neg_integer}
Metadata:
%{ adapter_meta: %{optional(atom) => term}, cluster_nodes: [node], reason: term }
telemetry_prefix ++ [:bootstrap, :joined]
- Dispatched by the adapter when the bootstrap has joined the cache to the cluster.Measurements:
%{system_time: non_neg_integer}
Metadata:
%{ adapter_meta: %{optional(atom) => term}, cluster_nodes: [node] }
Info API
As explained above, the partitioned adapter depends on the configured primary
storage adapter. Therefore, the information the info
command provides will
depend on the primary storage adapter. The Nebulex built-in adapters support
the recommended keys :server
, :memory
, and :stats
. Additionally, the
partitioned adapter supports:
:nodes_info
- A map with the info for each node.:nodes
- A list with the cluster nodes.
For example, the info for MyApp.PartitionedCache
may look like this:
iex> MyApp.PartitionedCache.info!()
%{
memory: %{total: nil, used: 344600},
server: %{
cache_module: MyApp.PartitionedCache,
cache_name: :partitioned_cache,
cache_adapter: Nebulex.Adapters.Partitioned,
cache_pid: #PID<0.1053.0>,
nbx_version: "3.0.0"
},
stats: %{
hits: 0,
misses: 0,
writes: 0,
evictions: 0,
expirations: 0,
deletions: 0,
updates: 0
},
nodes: [:"node1@127.0.0.1", ...],
nodes_info: %{
"node1@127.0.0.1": %{
memory: %{total: nil, used: 68920},
server: %{
cache_module: MyApp.PartitionedCache.Primary,
cache_name: MyApp.PartitionedCache.Primary,
cache_adapter: Nebulex.Adapters.Local,
cache_pid: #PID<23981.823.0>,
nbx_version: "3.0.0"
},
stats: %{
hits: 0,
misses: 0,
writes: 0,
evictions: 0,
expirations: 0,
deletions: 0,
updates: 0
}
},
...
}
}
Extended API
This adapter provides some additional convenience functions to the
Nebulex.Cache
API.
Retrieving the primary storage or local cache module:
MyCache.__primary__()
Retrieving the cluster nodes associated with the given cache name
:
MyCache.nodes()
Get a cluster node based on the given key
:
MyCache.find_node("mykey")
MyCache.find_node!("mykey")
Joining the cache to the cluster:
MyCache.join_cluster()
Leaving the cluster (removes the cache from the cluster):
MyCache.leave_cluster()
CAVEATS
For Nebulex.Cache.get_and_update/3
and Nebulex.Cache.update/4
,
they both have a parameter that is the anonymous function, and it is compiled
into the module where it is created, which means it necessarily doesn't exists
on remote nodes. To ensure they work as expected, you must provide functions
from modules existing in all nodes of the group.
Summary
Functions
Helper function to use dynamic cache for internal primary cache storage when needed.