Slap.Cluster.Strategy.Distributed (slap_cluster v0.2.0)

Copy Markdown View Source

Places shards on the connected nodes of a distributed Erlang cluster with no database and no leases: every node computes the same placement from the list of nodes running the cluster (a :pg group), by rendezvous hashing with bounded loads: shard n prefers the nodes in decreasing order of xxh64("<n>/<node>"), and goes to the first of them that has fewer than ceil(shards / nodes) shards so far. Every node gets its fair share, and a node joining or leaving moves few shards besides its own.

Every :interval (default 1 s), and whenever the member list changes, each node stops the shards it no longer owns (at once), and opens the shards it owns once that has been so for :settle ms (default 2 s), which gives the previous owner time to close them. lookup/2 reads the placement this node computed last. The strategy asks the host to open and close shards without waiting for replies, so a slow open or close does not hold up the rounds, and up to :max_concurrency shards open at once. All these requests come from the strategy's process, so the host gets them in order: a close requested while the shard opens is handled after the open (the host closes it once it has opened).

Failover follows distributed Erlang's view of the nodes: a node that dies or stops is gone at once, and its shards open elsewhere after :settle. A node that stops answering (a pause, a partition) is only noticed after net_ticktime (60 s by default; set it lower, for example -kernel net_ticktime 10).

Partitions. There is no quorum: in a partition each side computes a placement from the nodes it sees, and both may open a shard. SlateDB's fencing keeps at most one effective writer (the last to open fences the other, whose writes fail and are not acknowledged), so the cost is availability, not data. When the partition heals, :pg merges the member lists and every node computes the same placement again. For stronger guarantees, use ObjectLease.

Placement uses :pg membership because it resyncs after reconnection. :global names can retain split views after a node pauses and resumes.

Options

  • :interval - ms between placement rounds (default 1,000).
  • :settle - ms a node waits, after it becomes a shard's owner, before opening it (default 2,000).
  • :max_concurrency - shards opened at once (default 16).

Telemetry: [:slap, :cluster, :distributed, :open | :close] with %{cluster, shard}, and [:distributed, :members] with %{count: n} when the member list changes.

Summary

Functions

Returns a specification to start this module under a supervisor.

The placement of shards shards on nodes: %{shard => node}, the same on every node given the same nodes.

Functions

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

See Supervisor.

placement(shards, nodes)

@spec placement(pos_integer(), [node()]) :: %{required(non_neg_integer()) => node()}

The placement of shards shards on nodes: %{shard => node}, the same on every node given the same nodes.

start_link(opts)