Coyote-style controlled concurrency testing for BEAM.

See Lockstep.Test for the ExUnit integration. The functions in this module are the runtime API used inside controlled tests:

Lockstep.spawn(fn -> ... end)
Lockstep.send(target, message)
Lockstep.recv()

All three are sync points: each call hands control back to the controller, which picks the next process to run per the configured strategy.

Patterns and gotchas

Real-world usage of Lockstep — especially against distributed systems like Phoenix.PubSub, Phoenix.Tracker, or hand-rolled Raft-style protocols — surfaces a few patterns worth calling out up front.

Choosing a strategy

See Lockstep.Strategy for the full discussion. Short version: the default :pct strategy is best at finding partial-order races where any of several priority swaps work. For races that require the scheduler to consistently pick a specific proc over several consecutive sync points, :pos works better — PCT's priority shuffle under-explores those sequences. Real example from Lockstep's own test suite: test/leader_follower_register_test.exs async- replication staleness wasn't found in 100 PCT iterations but was found at iteration 1 under POS. Rule of thumb: if PCT can't find your race in ~50 iterations, try POS before increasing iterations.

Driving timed protocols: explicit triggers, not send_after

Distributed protocols often have time-driven elements: election timeouts, heartbeats, retry timers. There's a temptation to model them in tests with Lockstep.send_after, hoping the strategy will explore "what if the timeout fires before the response arrives" schedules. This usually doesn't work as intended.

Lockstep's controller fires timers ONLY when every alive proc is blocked on recv_match. So timer fires get serialized with proc execution: timer 1 fires → recipient becomes ready → strategy picks it → it processes → returns to recv → all blocked again → timer 2 fires. Multiple timers at the same fire_at don't actually fire concurrently — they're explored one at a time.

For tests where you want to surface concurrent-trigger races, drive the triggering events explicitly via Lockstep.send from the test body. test/raft_election_test.exs uses this pattern successfully:

# Trigger an election on every node concurrently. They'll all
# become candidates for term 1 and race for the majority.
for {_id, pid} <- nodes do
  Lockstep.spawn(fn ->
    Lockstep.send(pid, :trigger_election)
  end)
end

This way the strategy interleaves the per-node election handlers freely — the very interleaving the bug needs.

Avoiding timer pile-up

When a long-running test does need timers (heartbeats, retries), every handler that schedules a fresh timer should cancel or invalidate the previous one. Naive code:

def handle_info(:tick, state) do
  Lockstep.send_after(self(), :tick, 100)  # leak!
  {:noreply, work(state)}
end

Each tick adds another pending timer to the queue. After a few hundred ticks, virtual-time advancement is dominated by trivial timer fires that consume max_steps budget. Two clean fixes:

1. Cancel and re-schedule:

   def handle_info(:tick, state) do
     if state.timer, do: Lockstep.cancel_timer(state.timer)
     ref = Lockstep.send_after(self(), :tick, 100)
     {:noreply, %{state | timer: ref} |> work()}
   end

2. Epoch-tagged messages (no need to cancel; stale fires are ignored on receipt):

   def handle_info({:tick, epoch}, %{epoch: epoch} = state) do
     new_epoch = epoch + 1
     Lockstep.send_after(self(), {:tick, new_epoch}, 100)
     {:noreply, %{state | epoch: new_epoch} |> work()}
   end
   
   def handle_info({:tick, _stale}, state), do: {:noreply, state}

The Raft demo uses pattern #2.

Multi-step sync chains

Operations like Lockstep.GenServer.call/2 go through several sync points: a monitor, a send, and a selective receive. A test that performs N gen_server calls puts ~3N controller calls on each iteration's step budget. For long workloads or chatty libraries (Phoenix.Tracker is a notorious example — every heartbeat triggers ~10 sync points through Registry / persistent_term / ETS), max_steps may need to be set substantially higher than for simple race-hunt scenarios. Start with 5_000 for tight micro-races and scale to 50_000+ for full Phoenix integration tests.

v0.1 → v0.5 progression

v0.1 supported bare send/receive/spawn only. v0.5 added GenServer, Task, Registry, Supervisor, GenStatem wrappers, virtual clock, monitors, links, trap_exit. v1.0 (current) adds distributed- cluster simulation (Lockstep.Cluster), per-node state isolation (Phase D), and Jepsen-level checker infrastructure (Lockstep.History, Lockstep.Checker.{Linearizable, SequentialConsistency, Causal}, Lockstep.Generator, Lockstep.Model.Register).