Slap.Cluster.Strategy.ObjectLease (slap_cluster v0.2.0)

Copy Markdown View Source

Places shards on the nodes of a cluster through leases kept as objects in the cluster's own object store: no database is needed, only the bucket the shards already live in. It is the lease strategy for deployments without a database.

One object per shard, <path>/_cluster/placement/shards/NNN, holds {"owner", "expires_at", "generation"}, and one per node, .../nodes/<name>, its heartbeat. Claims and renewals are conditional writes: create-if-absent for a lease that does not exist yet, and If-Match on the version last read or written otherwise, so two nodes can never both claim or renew the same lease. The loop is Slap.Cluster.Strategy.LeaseLoop.

Routing (Slap.Cluster.call/4) still goes over distributed Erlang, to the node a lease names. That name must already be known on the caller's node (an atom: a node it is or was connected to, or one named in its configuration), since lease data does not create atoms; so the nodes need peer discovery, such as libcluster. Until then, a node's shards are unassigned from the nodes that do not know it.

Requirements

  • A store with conditional updates (If-Match): S3 and S3-compatible stores such as RustFS, Azure, GCS. The local file system only has create-if-absent, so it cannot be used (cluster startup fails); one node on a local directory should use Local.
  • Clocks within about :max_clock_skew_ms (default 1 s): expiry times are wall-clock times written by the owner and read by the others. A node only treats a lease as expired :max_clock_skew_ms after its expiry time. Beyond that, a claim can come early, which costs a fence in SlateDB, not data.

Cost

Every interval (a third of the TTL, 5 s by default) each node writes its heartbeat, lists and reads the heartbeats, renews its leases (one conditional PUT each), and reads every shard's lease: with 64 shards and 3 nodes, about 20 requests a second per node. A longer TTL costs less and fails over more slowly.

Options

  • :store - where the lease objects live (default the cluster's store).
  • :path - their path in it (default <cluster path>/_cluster/placement).
  • :max_clock_skew_ms - ms (default 1,000); see above.
  • the loop's options: :lease_ttl (default 15 s), :interval, :margin, :name, :cluster_name, :max_concurrency; see Slap.Cluster.Strategy.LeaseLoop.