Distributed locks on Erlang terms.

A lock is a term locked in a scope on a node. lock/4 takes it exclusively or shared, on one node or on several nodes at once, and returns a reference to release it with unlock/1. Waiting requests are served in order, the locks of a process are released when it exits, and deadlocks are detected, also across nodes and scopes.

This module is the whole public API. The other modules of the application are implementation details.

Setup

elock is an application: list it in the applications of your own application and it is started before yours. The nodes must be connected by Erlang distribution.

A scope is an independent set of locks named by an atom. Start it under a supervisor of yours, on every node that is to keep locks of the scope:

#{
  id => my_locks,
  start => {elock, start_link, [my_locks]}
}

Usage

Take a lock, do the work and release the lock in the same process:

{ok, Ref} = elock:lock(my_locks, {account, 42}, [node()]),
try
  withdraw(42, Amount)
after
  elock:unlock(Ref)
end.

A shared lock on every node of the scope, waiting for it at most 5 seconds:

Nodes = elock:ready_nodes(my_locks),
Options = #{is_shared => true, timeout => 5000},
case elock:lock(my_locks, {account, 42}, Nodes, Options) of
  {ok, Ref} ->
    Balance = balance(42),
    elock:unlock(Ref),
    {ok, Balance};
  {error, Reason} ->
    {error, Reason}
end.

Locks

• Any term can be locked. Terms are compared with =:=, so 1 and 1.0 are different locks.
• The same term in two scopes, or on two nodes, is two locks. Two requests compete only on the nodes that both of them name.
• An exclusive lock has one holder. A shared lock has any number of holders and keeps the exclusive requests waiting.
• The requests for a lock are served on each node in the order they arrive. A shared request joins the shared holders at once only while nobody waits, so a stream of shared requests does not starve an exclusive one.
• A lock belongs to the process that took it, and only that process can release it. The locks of a process are released and its waiting requests are withdrawn when it exits.
• A request for several nodes is all or nothing: it succeeds when every node has granted it, and when it fails the nodes that have granted it are released. While it waits for some nodes it holds the others.
• A failed request leaves the locks the process already holds untouched.

Re-entrancy

A process can lock a term it already holds. Every successful call returns a reference of its own, and the term is released when all of them are unlocked.

• A holder asking for the same mode is granted at once.
• An exclusive holder asking for shared is granted at once as well, and the lock stays exclusive until the exclusive reference is unlocked.
• A shared holder asking for exclusive is an upgrade. It waits for the other shared holders to leave and goes ahead of the queue. If two holders upgrade the same lock, the second gets {error, {deadlock, Lock}} at once and keeps its shared lock.

Deadlocks

Processes that wait for each other's locks in a cycle are detected, within a scope, across scopes and across nodes. At least one request of the cycle fails with {error, {deadlock, {Scope, Term, Node}}}, which names the lock the winning request waits for.

• A request of a younger context loses to a request of an older context. The context of a process starts with its first lock request while it holds nothing and ends when it releases its last lock. Equal ages are settled by a coin.
• A request can also fail with {error, {deadlock, probe_limit}}, without a cycle, when the search for one exceeds its limit. Handle it as a deadlock.
• The loser keeps the locks it holds, and the others keep waiting for them. To let them through, release those locks and then repeat the request.
• Two processes that lock the same term on the same several nodes at the same time may get a part of the nodes each. That is a deadlock as well, so a request for several nodes can lose while its process holds nothing.
• One cycle may cost more than one request: the verdicts are made independently, so several requests of a cycle issued at the same moment can lose, more often as the cycle gets longer.
• A request can lose to a cycle that has just dissolved. As after a real deadlock, the loser keeps its locks and the caller repeats the request.

Failures

• A lock lives on the node it was taken on. If that node goes down, or the scope stops there, the lock is lost and its holder is not notified. The locks the holder has on the other nodes stay until it unlocks them.
• A request fails as a whole when one of its nodes can not be reached or does not run the scope, see lock/4.