portunus (portunus v0.8.0)

View Source

The primary API of the portunus lock server. Its shape is inspired by ra's own API.

Lifecycle functions take a name() when they act on the local node (restart_server/2, join_or_form/3) and a server_id() when they can target any member's replica (reset_server/2).

Summary

Functions

Acquire LockKey under LeaseId, trying once. Returns {ok, Token}, the fencing token, or {error, {held_by, Owner}} if the key is held. It never queues: use acquire_or_join_succession_queue/4 to wait for the key.

Like acquire/4, attaching Context to the grant (returned by owner/2). Context is set-once and should be a pointer, not a payload: it lives in every replica's state, every snapshot and every owner/2 reply.

Acquire LockKey, or join its succession queue if it is held. Returns {ok, Token} if the key was free, otherwise {queued, Depth} (the number of waiters on the key, not a place in line), and later sends {portunus, granted, Key, Token, LeaseId} to the lease holder once it is promoted.

Like acquire_or_join_succession_queue/4, with #{affinity => Spec} to bias which contender is promoted first (see portunus_affinity; the default is FIFO in arrival order) and #{context => Term} to attach a context to the grant on promotion, as acquire/5 does for an immediate grant.

Acquire LockKey, waiting up to TimeoutMs for the current owner to release, be revoked, or expire.

Add a node as a member. Adding automatically, e.g. from a boot sequence, is safe; removal is never automatic (see remove_member/2).

The live contenders queued for LockKey, as their owner terms, read from the local replica. Non-blocking and approximate under replica lag, which suits the advisory transfer pre-check. Returns {error, no_quorum} if the local replica cannot answer.

Form (or join) a single cluster-wide portunus cluster from an Env map. Every member runs a replica.

Grant a lease with a TTL in milliseconds. See grant_lease/3 for options.

Grant a lease with a TTL in milliseconds. With #{auto_renew => true} a holder-linked renewer keeps it alive for as long as the calling process lives; the lease (and any locks held under it) ends when the caller dies or revokes. The returned id is used exactly like any other. A proposed_id makes the grant idempotent under retry; without one the id is auto-assigned. Auto-assigned ids are integers (Raft indices), so an integer proposed_id can collide with them and draw a spurious id_in_use; propose a tuple or any other non-integer term.

Whether the cluster currently has an online majority (a quorum-confirming read).

Whether this node is a member of cluster Name. Answered from the local replica without contacting the leader, so it holds during an election or a quorum loss, and is false when no local replica is running. Useful for a host that bootstraps the cluster and needs to know when this node has joined.

Start a local server and join the existing cluster that SeedNode belongs to. The local Ra system must already be running (start_system). Idempotent: succeeds if this node is already a member. As in start_cluster/3, the cluster Name is registered locally, so a collision with another registered process is reported as {error, {name_registered, Pid}}.

Attempts to join an existing portunus Raft cluster or create one.

Start a renewer that keeps LeaseId alive and is linked to the caller, so it stops when the caller does. Pass the TTL the lease was granted with; a renewer cannot sustain a TTL below 2000 ms (see portunus_keepalive). grant_lease/3 with #{auto_renew => true} does the same in one step.

Withdraw the lease's succession bid on LockKey: the opposite of joining the queue with acquire_or_join_succession_queue/4,5, for a contender that no longer wants the key. The current holder is untouched, and the lease's other keys and bids are unaffected. A lease with no bid on the key returns {error, not_queued}; a holder releases with release/3 instead.

One-shot exclusive lock with auto-renewal: grant a lease, acquire the key, and start a holder-linked renewer. Returns a handle for unlock/1.

Current members and the leader.

Query the owner of a lock (linearizable).

Release a held lock. Token-fenced: a stale token cannot release a re-granted lock.

Remove a member. Removal is explicit, never automatic.

Renew one or more leases in a single command (batch renew).

Renew with an explicit command timeout. A renewer bounds this to a fraction of the TTL so several attempts fit within one TTL across a leader change.

Reset this node's replica of cluster Name and join the cluster that SeedNode belongs to. Unlike join_cluster/3, this first wipes any existing local replica, so a node that formed its own single-node cluster during parallel boot can be merged into an existing cluster rather than colliding with it.

Stop and wipe a server's data dir. Refuses if still a member.

Restart this node's replica of cluster Name, recovering its state from disk. Use this after start_system/2, which rebuilds the Ra system.

Revoke a lease and release every lock held under it.

Start a portunus Ra cluster named Name across Nodes. Ra registers each node's server locally under Name, so Name must not collide with another registered process: a local collision is reported as {error, {name_registered, Pid}} rather than left to surface as cluster_not_formed.

Start a dedicated Ra system to use for portunus. Idempotent, and safe to call again after the ra application has been restarted.

A snapshot of cluster health and the machine-derived counts.

Hand a held lock to a chosen contender in one machine transition, keeping the key held by exactly one owner throughout. TargetOwner names the contender by its owner term; for the node-based batteries that owner is the node, so portunus_election:transfer_to/2 and portunus_registry:transfer/3 take a node().

Release a lock taken with lock/3: stop the renewer and revoke the lease.

Stop the watch registered under Ref.

Watch a key; the caller receives {portunus, watch, Ref, Event} messages, where Event is {acquired, Owner} or released. One watch per process per key: re-watching returns a new ref and supersedes the old. Pass Ref to unwatch/2.

Run Fun while holding LockKey, releasing on return or exception.

Types

acquire_error()

-type acquire_error() :: {held_by, owner()} | lease_expired | no_quorum.

acquisition_timeout_error()

-type acquisition_timeout_error() :: timeout | acquire_error().

env()

-type env() ::
          #{ra_system => system(),
            name => name(),
            data_dir => file:filename_all(),
            membership => [node()] | {module(), atom()} | local}.

grant_opts()

-type grant_opts() :: #{proposed_id => lease_id(), auto_renew => boolean()}.

handle()

-opaque handle()

lease_error()

-type lease_error() :: id_in_use | no_quorum.

lease_id()

-type lease_id() :: portunus_machine:lease_id().

leave_queue_error()

-type leave_queue_error() :: not_queued | no_quorum.

lock_key()

-type lock_key() :: portunus_machine:lock_key().

name()

-type name() :: atom().

ok_or_error(E)

-type ok_or_error(E) :: ok | {error, E}.

option(T)

-type option(T) :: T | undefined.

owner()

-type owner() :: portunus_machine:owner().

owner_info()

-type owner_info() :: portunus_machine:owner_info().

release_error()

-type release_error() :: not_owner | not_held | no_quorum.

server_id()

-type server_id() :: ra:server_id().

status()

-type status() ::
          #{leader := option(server_id()),
            members := [server_id()],
            quorum := boolean(),
            leases => non_neg_integer(),
            locks => non_neg_integer(),
            waiters => non_neg_integer(),
            watchers => non_neg_integer(),
            fencing_token => non_neg_integer()}.

succession_opts()

-type succession_opts() :: #{affinity => portunus_affinity:spec(), context => term()}.

system()

-type system() :: atom().

token()

-type token() :: portunus_machine:token().

transfer_error()

-type transfer_error() :: not_owner | {no_contender, owner()} | no_quorum.

ttl()

-type ttl() :: pos_integer().

watch_ref()

-type watch_ref() :: portunus_machine:watch_ref().

Functions

acquire(Name, LockKey, LeaseId, Owner)

-spec acquire(name(), lock_key(), lease_id(), owner()) -> {ok, token()} | {error, acquire_error()}.

Acquire LockKey under LeaseId, trying once. Returns {ok, Token}, the fencing token, or {error, {held_by, Owner}} if the key is held. It never queues: use acquire_or_join_succession_queue/4 to wait for the key.

acquire(Name, LockKey, LeaseId, Owner, Context)

-spec acquire(name(), lock_key(), lease_id(), owner(), term()) ->
                 {ok, token()} | {error, acquire_error()}.

Like acquire/4, attaching Context to the grant (returned by owner/2). Context is set-once and should be a pointer, not a payload: it lives in every replica's state, every snapshot and every owner/2 reply.

acquire_or_join_succession_queue(Name, LockKey, LeaseId, Owner)

-spec acquire_or_join_succession_queue(name(), lock_key(), lease_id(), owner()) ->
                                          {ok, token()} |
                                          {queued, pos_integer()} |
                                          {error, acquire_error()}.

Acquire LockKey, or join its succession queue if it is held. Returns {ok, Token} if the key was free, otherwise {queued, Depth} (the number of waiters on the key, not a place in line), and later sends {portunus, granted, Key, Token, LeaseId} to the lease holder once it is promoted.

acquire_or_join_succession_queue(Name, LockKey, LeaseId, Owner, Opts)

-spec acquire_or_join_succession_queue(name(), lock_key(), lease_id(), owner(), succession_opts()) ->
                                          {ok, token()} |
                                          {queued, pos_integer()} |
                                          {error, acquire_error()}.

Like acquire_or_join_succession_queue/4, with #{affinity => Spec} to bias which contender is promoted first (see portunus_affinity; the default is FIFO in arrival order) and #{context => Term} to attach a context to the grant on promotion, as acquire/5 does for an immediate grant.

acquire_with_timeout(Name, LockKey, LeaseId, Owner, TimeoutMs)

-spec acquire_with_timeout(name(), lock_key(), lease_id(), owner(), non_neg_integer()) ->
                              {ok, token()} | {error, acquisition_timeout_error()}.

Acquire LockKey, waiting up to TimeoutMs for the current owner to release, be revoked, or expire.

Must be called by the lease holder process, which receives the {portunus, granted, ...} message.

On timeout the bid is withdrawn, so the caller is never granted a key it gave up on. A grant that lands at the same instant is detected and the key is returned.

{error, no_quorum} on the timeout path means the withdrawal is unconfirmed: the bid may still be queued, so retry leave_succession_queue/3 or revoke the lease.

acquire_with_timeout(Name, LockKey, LeaseId, Owner, TimeoutMs, Opts)

-spec acquire_with_timeout(name(),
                           lock_key(),
                           lease_id(),
                           owner(),
                           non_neg_integer(),
                           succession_opts()) ->
                              {ok, token()} | {error, acquisition_timeout_error()}.

Like acquire_with_timeout/5, with the options of acquire_or_join_succession_queue/5.

add_member(Name, Node)

-spec add_member(name(), node()) -> ok_or_error(term()).

Add a node as a member. Adding automatically, e.g. from a boot sequence, is safe; removal is never automatic (see remove_member/2).

contenders(Name, LockKey)

-spec contenders(name(), lock_key()) -> {ok, [owner()]} | {error, no_quorum}.

The live contenders queued for LockKey, as their owner terms, read from the local replica. Non-blocking and approximate under replica lag, which suits the advisory transfer pre-check. Returns {error, no_quorum} if the local replica cannot answer.

effective_seed(Members, IsReachable)

-spec effective_seed([node()], fun((node()) -> boolean())) -> node().

ensure_started(Env)

-spec ensure_started(env()) -> {ok, [server_id()], [server_id()]} | {error, term()}.

Form (or join) a single cluster-wide portunus cluster from an Env map. Every member runs a replica.

This forms across all membership nodes at once, for a coordinated start where they come up together. For nodes that bootstrap independently, each running its own retry loop, use join_or_form/3 instead.

grant_lease(Name, TtlMs)

-spec grant_lease(name(), ttl()) -> {ok, lease_id()} | {error, lease_error()}.

Grant a lease with a TTL in milliseconds. See grant_lease/3 for options.

grant_lease/3

-spec grant_lease(name(), ttl(), grant_opts()) -> {ok, lease_id()} | {error, lease_error()}.

Grant a lease with a TTL in milliseconds. With #{auto_renew => true} a holder-linked renewer keeps it alive for as long as the calling process lives; the lease (and any locks held under it) ends when the caller dies or revokes. The returned id is used exactly like any other. A proposed_id makes the grant idempotent under retry; without one the id is auto-assigned. Auto-assigned ids are integers (Raft indices), so an integer proposed_id can collide with them and draw a spurious id_in_use; propose a tuple or any other non-integer term.

A holder that revokes an auto-renewed lease receives one final {portunus, lease_lost, LeaseId} when the renewer discovers the revocation; ignore it. For a renewer the holder can stop explicitly, use keep_alive/3, which returns the renewer's pid.

has_quorum(Name)

-spec has_quorum(name()) -> boolean().

Whether the cluster currently has an online majority (a quorum-confirming read).

is_member(Name)

-spec is_member(name()) -> boolean().

Whether this node is a member of cluster Name. Answered from the local replica without contacting the leader, so it holds during an election or a quorum loss, and is false when no local replica is running. Useful for a host that bootstraps the cluster and needs to know when this node has joined.

A local view alone cannot decide this: Ra always includes a server in its own view, joined or not. A genuine member also holds at least the log entry that added it, so an empty log means the join never committed.

join_cluster(System, Name, SeedNode)

-spec join_cluster(system(), name(), node()) -> ok_or_error(term()).

Start a local server and join the existing cluster that SeedNode belongs to. The local Ra system must already be running (start_system). Idempotent: succeeds if this node is already a member. As in start_cluster/3, the cluster Name is registered locally, so a collision with another registered process is reported as {error, {name_registered, Pid}}.

join_or_form(System, Name, Members)

-spec join_or_form(system(), name(), [node()]) -> ok_or_error(term()).

Attempts to join an existing portunus Raft cluster or create one.

The algorithm is based on the concept of a seed member (cluster node) that is computed as the lowest (in lexicographical order) reachable node on the Members list. The Members list MUST include the current node: when using the return value of erlang:nodes/0, don't forget to prepend the value of erlang:node/0 as well.

The seed node is asynchronously and independently agreed on by different members, not pre-selected or pre-computed before deployment time.

When a new cluster is formed and a member computes that it itself is the seed, it kicks off a Raft election. This is true for both single-node and multi-node clusters.

Most designs that calculate a seed node suffer from that node not being available. In order to avoid this, portunus picks the lowest (first) reachable member and if that turns out to be a different member, (re)joins that member's existing Raft cluster.

If the member (node) has existing on-disk member data and the computed seed responds that the asking (calling) member is not on its known cluster member list, such node will reset itself and join the seed's Raft cluster.

When the seed member PS is restarted and finds its pre-existing member data, it will also compute the seed and should the seed node have changed during the PS absence, join the new seed's cluster. If the existing on-disk data includes the new seed NS, the rejoining PS will not reset itself and instead will catch up with the current Raft leader.

However, it's important to explain when the reset happens. The newly joining member will clarify its membership with the seed and only then perform the reset and join the seed.

The companion function that initiates a coordinated start is ensure_started/1.

If the seed is unreachable, returns an error (such as {error, seed_unreachable}) and does not reset the local member.

keep_alive(Name, LeaseId, TtlMs)

-spec keep_alive(name(), lease_id(), ttl()) -> {ok, pid()}.

Start a renewer that keeps LeaseId alive and is linked to the caller, so it stops when the caller does. Pass the TTL the lease was granted with; a renewer cannot sustain a TTL below 2000 ms (see portunus_keepalive). grant_lease/3 with #{auto_renew => true} does the same in one step.

leave_succession_queue(Name, LockKey, LeaseId)

-spec leave_succession_queue(name(), lock_key(), lease_id()) -> ok_or_error(leave_queue_error()).

Withdraw the lease's succession bid on LockKey: the opposite of joining the queue with acquire_or_join_succession_queue/4,5, for a contender that no longer wants the key. The current holder is untouched, and the lease's other keys and bids are unaffected. A lease with no bid on the key returns {error, not_queued}; a holder releases with release/3 instead.

lock(Name, LockKey, TtlMs)

-spec lock(name(), lock_key(), ttl()) -> {ok, handle()} | {error, acquire_error() | lease_error()}.

One-shot exclusive lock with auto-renewal: grant a lease, acquire the key, and start a holder-linked renewer. Returns a handle for unlock/1.

members(Name)

-spec members(name()) -> {ok, [server_id()], server_id()} | {error, term()}.

Current members and the leader.

owner(Name, LockKey)

-spec owner(name(), lock_key()) -> {ok, owner_info()} | {error, not_held | no_quorum}.

Query the owner of a lock (linearizable).

release(Name, LockKey, Token)

-spec release(name(), lock_key(), token()) -> ok_or_error(release_error()).

Release a held lock. Token-fenced: a stale token cannot release a re-granted lock.

remove_member(Name, Node)

-spec remove_member(name(), node()) -> ok_or_error(term()).

Remove a member. Removal is explicit, never automatic.

renew_leases(Name, LeaseIds)

-spec renew_leases(name(), [lease_id()]) -> [{lease_id(), ok | {error, lease_expired | no_quorum}}].

Renew one or more leases in a single command (batch renew).

renew_leases(Name, LeaseIds, Timeout)

-spec renew_leases(name(), [lease_id()], timeout()) ->
                      [{lease_id(), ok | {error, lease_expired | no_quorum}}].

Renew with an explicit command timeout. A renewer bounds this to a fraction of the TTL so several attempts fit within one TTL across a leader change.

reset_and_join_cluster(System, Name, SeedNode)

-spec reset_and_join_cluster(system(), name(), node()) -> ok_or_error(term()).

Reset this node's replica of cluster Name and join the cluster that SeedNode belongs to. Unlike join_cluster/3, this first wipes any existing local replica, so a node that formed its own single-node cluster during parallel boot can be merged into an existing cluster rather than colliding with it.

The local replica's state is discarded, so call this during cluster formation, before the node holds any leases worth keeping.

reset_server/2

-spec reset_server(system(), server_id()) -> ok_or_error(still_member | no_quorum).

Stop and wipe a server's data dir. Refuses if still a member.

restart_server(System, Name)

-spec restart_server(system(), name()) -> ok_or_error(term()).

Restart this node's replica of cluster Name, recovering its state from disk. Use this after start_system/2, which rebuilds the Ra system.

Returns {error, name_not_registered} if this node has no on-disk member data for Name, in which case it was never a member and should use start_cluster/3 or join_cluster/3 instead.

revoke_lease(Name, LeaseId)

-spec revoke_lease(name(), lease_id()) -> ok_or_error(no_quorum).

Revoke a lease and release every lock held under it.

settle_timed_out_bid(Name, LockKey, LeaseId)

-spec settle_timed_out_bid(name(), lock_key(), lease_id()) ->
                              {ok, token()} | {error, timeout | no_quorum}.

start_cluster(System, Name, Nodes)

-spec start_cluster(system(), name(), [node()]) -> {ok, [server_id()], [server_id()]} | {error, term()}.

Start a portunus Ra cluster named Name across Nodes. Ra registers each node's server locally under Name, so Name must not collide with another registered process: a local collision is reported as {error, {name_registered, Pid}} rather than left to surface as cluster_not_formed.

start_system(System, DataDir)

-spec start_system(system(), file:filename_all()) -> ok_or_error(term()).

Start a dedicated Ra system to use for portunus. Idempotent, and safe to call again after the ra application has been restarted.

status(Name)

-spec status(name()) -> status().

A snapshot of cluster health and the machine-derived counts.

transfer(Name, LockKey, Token, TargetOwner)

-spec transfer(name(), lock_key(), token(), owner()) -> ok_or_error(transfer_error()).

Hand a held lock to a chosen contender in one machine transition, keeping the key held by exactly one owner throughout. TargetOwner names the contender by its owner term; for the node-based batteries that owner is the node, so portunus_election:transfer_to/2 and portunus_registry:transfer/3 take a node().

Token-fenced like release/3: only the current holder can transfer, and a stale token or a free key returns {error, not_owner}. A TargetOwner equal to the holder's own owner returns ok. If no live contender carries TargetOwner the holder keeps the key and the reply is {error, {no_contender, TargetOwner}}, never a release to some other node.

unlock/1

-spec unlock(handle()) -> ok.

Release a lock taken with lock/3: stop the renewer and revoke the lease.

unwatch(Name, Ref)

-spec unwatch(name(), watch_ref()) -> ok_or_error(no_quorum).

Stop the watch registered under Ref.

watch(Name, LockKey)

-spec watch(name(), lock_key()) -> {ok, watch_ref()} | {error, no_quorum}.

Watch a key; the caller receives {portunus, watch, Ref, Event} messages, where Event is {acquired, Owner} or released. One watch per process per key: re-watching returns a new ref and supersedes the old. Pass Ref to unwatch/2.

with_lock(Name, LockKey, TtlMs, Fun)

-spec with_lock(name(), lock_key(), ttl(), fun(() -> Result)) ->
                   Result | {error, acquire_error() | lease_error()}.

Run Fun while holding LockKey, releasing on return or exception.