The placement loop of Slap.Cluster.Strategy.ObjectLease (leases in the object store), behind a storage behaviour for other lease stores: one lease per shard, a heartbeat per node, fair shares, and self-fencing. Where the leases live is a Slap.Cluster.Strategy.LeaseLoop.Store.

Every :interval (a third of :lease_ttl by default), each node:

1. heartbeats;
2. renews its leases, and stops any shard whose lease it no longer holds;
3. computes its fair share, ceil(shards / live nodes), where a node is live if it heartbeated within the TTL (three heartbeats), so a dead node stops counting when its leases expire;
4. releases shards above its share: it stops the shard (which closes and flushes it) and only then frees the lease, so the next owner opens a closed database;
5. reads every lease (for lookup/2), and claims free or expired ones up to its share, opening those shards with the lease's generation.

Shards open and close in a process of their own, the runner, so renewals keep their pace however long a rebalance takes. A shard keeps its lease (and it is renewed) while it opens or closes. After a failed renewal the loop tries again after half the interval.

Self-fencing. The runner (not the loop, which may be stuck in a slow call to the lease store) keeps a deadline per shard: its last successful renewal or claim, plus the TTL, minus :margin. A shard whose deadline passes is stopped before its lease can expire, and one still waiting to open (:max_concurrency are opened at once) is not opened. Every open and close goes to the host from the runner, so the host gets each shard's in order: a stop always comes after the open it follows. If the runner exits, the loop stops too, and the cluster's supervisor stops every shard. If a node is too slow even for that (a long pause), the new owner's open fences it in SlateDB, so there is still at most one effective writer.

A failed read of the owners skips the claims of that round, and lookup/2 keeps the owners of the last successful read. An owner is routed to only if its name is already an atom on this node (it is connected, was connected, or is named in configuration such as a peer discovery topology): lease data never creates atoms. A node that has never heard of an owner treats its shards as unassigned until the nodes connect, so the nodes need peer discovery (for example libcluster).

Options common to the lease strategies: :lease_ttl (ms, default 15,000), :interval (default a third of the TTL), :margin (default a sixth of the TTL), :name (this node's name, default node(); call/4 routes to it), :cluster_name (default the cluster module's name), and :max_concurrency (shards opened at once, default 16).

Telemetry: [:slap, :cluster, :lease, :claim | :release | :lost] with %{cluster, shard}, [:lease, :self_fence] with %{cluster, shards}, and [:lease, :renew_failed].