partisan_plumtree_broadcast (partisan v5.0.1)
View SourceThis module implements the Plumtree Protocol which is the component that materialises Partisan's gossip scheme.
The component is implemented as as a gen_server
that has two functions:
- Tree construction
- This component is in charge of selecting which links of the random overlay network will be used to forward the message payload using an eager push strategy. It implements a tree construction mechanisms that is as simple as possible, with minimal overhead in terms of control messages.
- Tree repair
- This component is in charge of repairing the tree when failures occur. The process ensures that, despite failures, all nodes remain covered by the spanning tree. Therefore, it should be able to detect and heal partitions of the tree. The overhead imposed by this operation should also be as low as possible.
Overview
The protocol operates as any pure gossip protocol, in the sense that, in order to broadcast a message, each node gossips with f
nodes provided by a peer sampling service (where f
is the protocol fanout). However, each node uses a combination of eager push and lazy push gossip.
Eager push is used just for a subset of the f
nodes, while lazy push is used for the remaining nodes. The links used for eager push are selected in such a way that their closure effectively builds a broadcast tree embedded in the random overlay network. Lazy push links are used to ensure gossip reliability when nodes fail and also to quickly heal the broadcast tree. Furthermore, the set of (random) peers is not changed at each gossip round. Instead, the same peers are used until failures are detected.
The protocol uses a Partisan Channel (See {link broadcast_channel}), i.e. based on TCP, to support the message exchange, as they offer extra reliability and an additional source of failure detection.
Plumtree depends on an overlay network which is maintained by a peer sampling service (partisan_peer_service_manager
) and relies on the peer sampling services/s exhibiting a property: Symmetric (partial) views. If the links that form the spanning tree are symmetric, then the tree may be shared by multiple sources. Symmetric partial views render the task of creating bi-directional trees easier, and reduce the amount of peers that each node has to maintain.
Summary
Types
Lazy messages that have not been acked. Messages are added to this set when a node is sent a lazy message (or when it should be sent one sometime in the future). Messages are removed when the lazy pushes are acknowledged via graft or ignores. Entries are keyed by their destination These are stored in the ?PLUMTREE_OUTSTANDING ets table under using nodename as key. PLUMTREE_OUTSTANDING is created and owned by partisan_sup
Functions
Broadcasts a message originating from this node. The message will be delivered to each node at least once. The Mod
passed must be loaded on all members of the cluster and implement the partisan_plumtree_broadcast_handler
behaviour which is responsible for handling the message on remote nodes as well as providing some other information both locally and on other nodes.
Returns the channel to be used when sending broadcasting a message on behalf of module Mod
.
Returns the broadcast servers view of full cluster membership. Wait indefinitely for a response is returned from the process.
Returns the broadcast servers view of full cluster membership. Waits Timeout
ms for a response from the server.
Cancel exchanges started by this node.
return the peers for Node
for the tree rooted at Root
. Wait indefinitely for a response is returned from the process
return the peers for Node
for the tree rooted at Root
. Waits Timeout
ms for a response from the server
return the peers for Node
for the tree rooted at Root
. Waits Timeout
ms for a response from the server
return peers for all Nodes
for tree rooted at Root
Wait indefinitely for a response is returned from the process
return peers for all Nodes
for tree rooted at Root
Wait Timeout
for a response is returned from the process
return peers for all Nodes
for tree rooted at Root
Wait Timeout
for a response is returned from the process
return a list of exchanges, started by broadcast on this node, that are running.
Returns a list of running exchanges, started on Node
.
Returns a list of running exchanges, started on Node
.
picks random peer favoring peers not in eager or lazy set and ensuring peer is not this node
Starts the broadcast server on this node.
Starts the broadcast server on this node. Members
must be a list of all members known to this node when starting the broadcast server. Eagers
are the initial peers of this node for all broadcast trees. Lazys
is a list of random peers not in Eagers
that will be used as the initial lazy peer shared by all trees for this node. If the number of nodes in the cluster is less than 3, Lazys
should be an empty list. Eagers
and Lazys
must also be subsets of Members
. Mods
is a list of modules that may be handlers for broadcasted messages. All modules in Mods
should implement the partisan_plumtree_broadcast_handler
behaviour.
Notifies broadcast server of membership update This is the function is added to partisan_peer_service_events using partisan_peer_service:add_sup_callback(fun ?MODULE:update/1),
Types
-type exchanges() :: [exchange()].
-type info_opt() :: node_spec | metadata | distance.
-type message_id() :: any().
-type message_round() :: non_neg_integer().
Lazy messages that have not been acked. Messages are added to this set when a node is sent a lazy message (or when it should be sent one sometime in the future). Messages are removed when the lazy pushes are acknowledged via graft or ignores. Entries are keyed by their destination These are stored in the ?PLUMTREE_OUTSTANDING ets table under using nodename as key. PLUMTREE_OUTSTANDING is created and owned by partisan_sup
-type nodeset() :: ordsets:ordset(node()).
-type opts_list() :: [{lazy_tick_period, non_neg_integer()} | {exchange_tick_period, non_neg_integer()}].
-type opts_map() :: #{lazy_tick_period => non_neg_integer(), exchange_tick_period => non_neg_integer()}.
-type outstanding() :: {message_id(), module(), message_round(), node()}.
-type state() :: #state{node :: node(), common_eagers :: nodeset(), common_lazys :: nodeset(), eager_sets :: #{node() := nodeset()}, lazy_sets :: #{node() := nodeset()}, mods :: [module()], exchanges :: exchanges(), all_members :: nodeset(), lazy_tick_period :: non_neg_integer(), exchange_tick_period :: non_neg_integer()}.
Functions
Broadcasts a message originating from this node. The message will be delivered to each node at least once. The Mod
passed must be loaded on all members of the cluster and implement the partisan_plumtree_broadcast_handler
behaviour which is responsible for handling the message on remote nodes as well as providing some other information both locally and on other nodes.
The broadcast will be sent over the channel defined by broadcast_channel/1
.
-spec broadcast_channel(Mod :: module()) -> partisan:channel().
Returns the channel to be used when sending broadcasting a message on behalf of module Mod
.
The channel defined by the callback Mod:broadcast_channel()
or default channel i.e. partisan:default_channel/0
if the callback is not implemented.
-spec broadcast_members() -> nodeset().
Returns the broadcast servers view of full cluster membership. Wait indefinitely for a response is returned from the process.
-spec broadcast_members(infinity | pos_integer()) -> nodeset().
Returns the broadcast servers view of full cluster membership. Waits Timeout
ms for a response from the server.
Cancel exchanges started by this node.
return the peers for Node
for the tree rooted at Root
. Wait indefinitely for a response is returned from the process
-spec debug_get_peers(node(), node(), infinity | pos_integer()) -> {nodeset(), nodeset()} | no_return().
return the peers for Node
for the tree rooted at Root
. Waits Timeout
ms for a response from the server
-spec debug_get_peers(node(), node(), [info_opt()], infinity | pos_integer()) -> {nodeset(), nodeset(), Info :: map()} | no_return().
return the peers for Node
for the tree rooted at Root
. Waits Timeout
ms for a response from the server
return peers for all Nodes
for tree rooted at Root
Wait indefinitely for a response is returned from the process
return peers for all Nodes
for tree rooted at Root
Wait Timeout
for a response is returned from the process
-spec debug_get_tree(node(), [node()], [info_opt()], timeout()) -> [{node(), {nodeset(), nodeset(), Info :: map()} | down}].
return peers for all Nodes
for tree rooted at Root
Wait Timeout
for a response is returned from the process
-spec exchanges() -> {ok, exchanges()}.
return a list of exchanges, started by broadcast on this node, that are running.
Returns a list of running exchanges, started on Node
.
Returns a list of running exchanges, started on Node
.
-spec get_peers(Root :: node(), Opts :: [partisan:info_opt()]) -> list().
picks random peer favoring peers not in eager or lazy set and ensuring peer is not this node
Starts the broadcast server on this node.
The initial membership list is fetched from the configured {link partisan_peer_service}.
If the node is a singleton then the initial eager and lazy sets are empty. If there are two nodes, each will be in the others eager set and the lazy sets will be empty. When number of members is less than 5, each node will initially have one other node in its eager set and lazy set. If there are more than five nodes each node will have at most two other nodes in its eager set and one in its lazy set, initially.
In addition, after the broadcast server is started, all callbacks defined in the configuration option broadcast_mods
are registered. By default the list of callbacks includes the module partisan_plumtree_backend
which is used by to generate membership updates as the ring changes.
-spec start_link(Members :: [node()], Eagers :: [node()], Lazys :: [node()], Mods :: [module()], Opts :: opts()) -> {ok, pid()} | ignore | {error, term()}.
Starts the broadcast server on this node. Members
must be a list of all members known to this node when starting the broadcast server. Eagers
are the initial peers of this node for all broadcast trees. Lazys
is a list of random peers not in Eagers
that will be used as the initial lazy peer shared by all trees for this node. If the number of nodes in the cluster is less than 3, Lazys
should be an empty list. Eagers
and Lazys
must also be subsets of Members
. Mods
is a list of modules that may be handlers for broadcasted messages. All modules in Mods
should implement the partisan_plumtree_broadcast_handler
behaviour.
Opts
is a proplist or map with the following possible options:
lazy_tick_period :: non_neg_integer()
- Flush all outstanding lazy pushes period (in milliseconds)exchange_tick_period :: non_neg_integer()
- Possibly perform an exchange period (in milliseconds)
NOTE: When starting the server using start_link/2 no automatic membership update from ring_events is registered. Use start_link/0
.
-spec update([node()]) -> ok.
Notifies broadcast server of membership update This is the function is added to partisan_peer_service_events using partisan_peer_service:add_sup_callback(fun ?MODULE:update/1),