The queues of a cluster's nodes. Each node delivers the mail in its
own spool (queue.directory); this process keeps copies of it on
other nodes, hands mail to the node that should send it, and hands a
node's queue to the others when it leaves.
Copies, and taking over from a node that is down
With cluster.queue_replicas (1 by default, up to 2), each message a
node queues is copied to that many other nodes, picked by the
message's queue ID among those connected, and copied again whenever
it is deferred or held, so that the copy has its delivery records.
When the message is done (delivered, bounced, expired, or deleted),
the copies are deleted.
When a node stays down for cluster.handover_after (10 minutes), the
first of its message's copy holders that is still up queues the copy
and delivers it, under the same queue ID, and tells the others to
delete theirs. When the node comes back, it is told which of its
messages were taken over, and deletes them.
Mail is delivered at least once, as SMTP allows, but in some cases
more than once: when a node is cut off from the others without being
down, or comes back while taking over is under way, both deliver; and
a copy that missed its last update is delivered again to the
recipients it had pending then. A long handover_after makes this
rare. Messages queued while a node had no peer have no copies.
Copies are in <queue.directory>/cluster/, outside the queues.
Handing mail to another node
A message's recipients for a destination go to another node, as a new message, when:
- their IP pool (
Sovite.Core.IPPools) has no address on this node: to a node that has some; - with
cluster.destination_affinity, another node owns their destination (MX domain or relay host): each destination is owned by one node, picked by its name among those connected, so all the cluster's mail for it leaves from that node, which keeps fewer connections to it busier.
The new message, with only those recipients and with forwarded_from
set, is delivered by that node, which never hands it on. The
recipients count as delivered here, without a success notification:
the other node sends those.
Leaving
Sovite.Core.Cluster.leave/1 (sovitectl cluster leave) drains this
node, and hands every queued message to the connected nodes
(hand_over_all/1), before it takes the node out of the cluster.
Telemetry
[:sovite, :cluster, :queue, :taken_over]-%{messages},%{node}, when this node queues the copies of a node that is down.[:sovite, :cluster, :queue, :handed_over]-%{messages, failed},%{}, when this node leaves.
Summary
Functions
Returns a specification to start this module under a supervisor.
Hands every queued message to the connected nodes, for a node that
leaves (see Sovite.Core.Cluster.leave/1, which drains it first).
Returns how many messages were handed over, and how many could not be
(they stay here).
The copies this node holds, by node, and the messages it took over and has to report.
Functions
Returns a specification to start this module under a supervisor.
See Supervisor.
@spec hand_over_all(GenServer.server()) :: {:ok, %{handed_over: non_neg_integer(), failed: non_neg_integer()}} | {:error, :no_peers}
Hands every queued message to the connected nodes, for a node that
leaves (see Sovite.Core.Cluster.leave/1, which drains it first).
Returns how many messages were handed over, and how many could not be
(they stay here).
@spec status(GenServer.server()) :: map()
The copies this node holds, by node, and the messages it took over and has to report.