Nebulex v1.2.0 Nebulex.Adapters.Partitioned View Source

Built-in adapter for partitioned cache topology.

A partitioned cache is a clustered, fault-tolerant cache that has linear scalability. Data is partitioned among all the machines of the cluster. For fault-tolerance, partitioned caches can be configured to keep each piece of data on one or more unique machines within a cluster. This adapter in particular hasn't fault-tolerance built-in, each piece of data is kept in a single node/machine (sharding), therefore, if a node fails, the data kept by this node won't be available for the rest of the cluster.

This adapter depends on a local cache adapter (primary storage), it adds a thin layer on top of it in order to distribute requests across a group of nodes, where is supposed the local cache is running already.

PG2 is used under-the-hood by the adapter to manage the cluster nodes. When the partitioned cache is started in a node, it creates a PG2 group and joins it (the cache supervisor PID is joined to the group). Then, when a function is invoked, the adapter picks a node from the node list (using the PG2 group members), and then the function is executed on that node. In the same way, when the supervisor process of the partitioned cache dies, the PID of that process is automatically removed from the PG2 group; this is why it's recommended to use a consistent hashing algorithm for the node selector.

Features

  • Support for partitioned topology (Sharding Distribution Model)
  • Support for transactions via Erlang global name registration facility
  • Configurable hash-slot module to compute the node

Options

These options can be set through the config file:

  • :primary - The module for the primary storage. The value must be a valid local cache adapter so that the partitioned adapter can store the data in there. For example, you can set the Nebulex.Adapters.Local as value, unless you want to provide another one.

  • :hash_slot - The module that implements Nebulex.Adapter.HashSlot behaviour.

Runtime options

These options apply to all adapter's functions.

  • :timeout - The time-out value in milliseconds for the command that will be executed. If the timeout is exceeded, then the current process will exit. This adapter uses Task.await/2 internally, therefore, check the function documentation to learn more about it. For commands like set_many and get_many, if the timeout is exceeded, the task is shutted down but the current process doesn't exit, only the result associated to that task is just skipped in the reduce phase.

  • task_supervisor_opts - Defines the options passed to Task.Supervisor.start_link/1 when the adapter is initialized.

Example

Nebulex.Cache is the wrapper around the cache. We can define the partitioned cache as follows:

defmodule MyApp.PartitionedCache do
  use Nebulex.Cache,
    otp_app: :my_app,
    adapter: Nebulex.Adapters.Partitioned

  defmodule Primary do
    use Nebulex.Cache,
      otp_app: :my_app,
      adapter: Nebulex.Adapters.Local
  end
end

Where the configuration for the cache must be in your application environment, usually defined in your config/config.exs:

config :my_app, MyApp.PartitionedCache.Primary,
  n_shards: 2,
  gc_interval: 3600

config :my_app, MyApp.PartitionedCache,
  primary: MyApp.PartitionedCache.Primary

For more information about the usage, check out Nebulex.Cache.

Extended API

This adapter provides some additional functions to the Nebulex.Cache API.

__primary__

Returns the local cache adapter (the local backend).

__task_sup__

Returns the task supervisor module that manages RPC calls.

__nodes__

Returns the nodes that belongs to the caller Cache.

get_node/1

This function invokes c:Nebulex.Adapter.NodeSelector.get_node/2 internally.

MyCache.get_node("mykey")

Limitations

This adapter has a limitation for two functions: get_and_update/4 and update/5. They both have a parameter that is the anonymous function, and the anonymous function is compiled into the module where it is created, which means it necessarily doesn't exists on remote nodes. To ensure they work as expected, you must provide functions from modules existing in all nodes of the group.

Link to this section Summary

Functions

Helper to perform stream/3 locally.

Link to this section Functions

Link to this function

eval_local_stream(cache, query, opts)

View Source

Helper to perform stream/3 locally.