-spec attach(term(), [dest()]) -> ok.

Puts processes on a simulated node without putting them on the wire: they receive the node's link events and die with it, and their traffic is never faulted.

eta_net:place(node_a, [MemberA]),          %% talks to peers over the network
eta_net:attach(node_a, [ElectorA, ConnA]). %% on the node, off the wire

This is the honest way to model a process whose "messages" are not messages. An elector that coordinates through a durable store, a connector that only watches node liveness — dropping their sends injects a failure the real system cannot have, and the divergence that follows is an artefact that reads exactly like a replication defect. But both must still learn when their node's peers go away, and both must die when the node does, so leaving them unplaced is wrong too.

Faultability is per process, not per channel: a send is faulted only when both ends are faultable, so an attached process is safe in either direction.

Otherwise identical to place/2 — same node names, same inheritance, same participation in partition/3 and kill_node/2.

-spec call(dest() | term(), term()) -> term().

gen_server:call/2,3 and gen_statem:call/2,3, with both legs on the network. One target for both, because the request is the same {'$gen_call', From, Request} either way.

The request is an ordinary send this module routes. The reply cannot be reached from here — gen:reply sends it from inside OTP — so eta_transform brings it on from the other end, rewriting the callee's returns: a gen_server's {reply, R, S} becomes eta_net:reply(From, R) plus {noreply, S}, and a gen_statem's {reply, From, R} action is taken out of the action list and sent through here instead. See statem_return/1.

Both legs can therefore be dropped or delayed independently, which is what makes the asymmetric fault reachable: the work happened, the caller never learned it did.

Monitors the callee and takes a timeout, as gen_server:call/3 does, with two differences:

• The timeout is virtual, so waiting one out on a dropped request costs no real time and fires where the schedule chose. See below for when.
• The monitor is eta_net's own and is deliberately left real, rather than going through monitor/2. It exists to notice a callee that died, and it has to work whatever the scheduler is or is not reporting — this is the path that turns a dead callee into an exit instead of a hang. The cost is that a call outstanding across a partition/3 ends in its (virtual) timeout rather than in noconnection, which is a known simplification. Monitors the system holds are simulated and do get noconnection; see monitor/2.

Raises unrouted_reply if the callee was not built with the transform, since its reply then comes around the network and one direction of the channel is silently unfaultable. Only on the routed path — with no network there is nothing to route and nothing to come around.

Three paths, and the timeout is virtual on two of them

The network is not what decides whether the timeout is real. The clock is, which is a distinction that used to be got wrong here and is worth stating plainly, because the default run (net => false) took the wrong branch.

• Network running — the request is routed, and the wait is on the virtual clock.
• No network, but a virtual clock — the request is an ordinary send and the wait is still on the virtual clock. This is gen:call's own protocol with the deadline moved, alias and all, so a reply that loses the race to the timeout is dropped by the VM exactly as it is outside a simulation.
• Neither — a plain gen_server:call/3, real timeout, which is what the module does about everything when it is inert.

What made the old gating a bug rather than a gap: a transformed system with the scheduler and the clock up and no network is the ordinary configuration, and every gen_server:call in it was waiting out a real 5-second after inside OTP while the driver did other work. When it fired was decided by machine load, so a seed did not reproduce its own run, and nothing reported it.

A virtual deadline is only armed for a process the scheduler is stepping. eta_run advances the clock only to deadlines belonging to a process it owns and steps over every other one as a stray (eta_time:advance_to_next/1), so a virtual timeout armed by anything else — the driver, a harness execute/2 that calls instead of spawning — is a deadline nothing will ever reach, and a hang with no timeout to end it. Those callers keep the real one. The check is a read of eta_sched:stepping/0, and it fails towards the real timeout: an owned caller this misses gets nondeterminism, which is bad, where an unowned caller it wrongly claimed would get a stalled run, which is worse.

-spec cast(dest() | term(), term()) -> ok.

gen_server:cast/2, routed. Never raises, including for an unregistered name.

-spec cut(dest(), dest()) -> ok.

Drops everything sent from From to To until heal/2. One direction.

Messages already in flight on that channel are cancelled, as a failing link loses what was on it, and counted as both dropped and cancelled.

No link events. This is a message fault, not a node failure: nothing is signalled and no monitor fires. A one-way loss of messages is not something a lost connection produces — real distribution tears the whole connection down when either end's tick times out, and both ends then find out. partition/3 is that event; cut/2 is the narrower fault of a channel that swallows traffic while both ends still believe the link is up. Use it when that is what you mean, and partition/3 when a node has gone.

-spec demonitor(reference()) -> true.

erlang:demonitor/1. See monitor/2.

-spec demonitor(reference(), list()) -> boolean().

erlang:demonitor/2, including flush and info. See monitor/2.

A simulated monitor that has already fired behaves as a real fired one does: true, or false under info, and never a raise.

-spec drop_matching(dest(), dest(), term(), non_neg_integer(), non_neg_integer()) -> ok.

On the From -> To channel, lets the next Skip messages tagged Tag through, then drops the following K of them. Other traffic is untouched.

Counts within one kind of message, which a raw message count cannot: a batch interleaves on the wire with acks and replies, so "drop messages 2..4 of this batch" is only expressible by tag. Losing a strict subset of one batch is a materially different fault from losing a whole channel — it can leave no discontinuity for the receiver to detect.

See set_policy/1 for what a tag is.

-spec drop_next(dest(), dest(), non_neg_integer()) -> ok.

Drops exactly the next K messages from From to To, then resumes. Asks what a probability cannot: "lose the reply to this one operation".

-spec faultable(dest()) -> boolean().

Whether the network may fault this process's traffic.

false for an unplaced process and for one added with attach/2; true for a placed one, and for everything when no topology has been declared at all. See place/2.

-spec heal(dest(), dest()) -> ok.

Restores a channel previously cut/2.

-spec heal_all() -> ok.

Removes every cut and every pending drop_next/3 or drop_matching/5.

Does not touch the random policy — heal_all/0 plus set_policy(#{drop_p => 0.0, delay_p => 0.0}) is the perfect network to converge into.

Does not touch monitors or topology either. A monitor retired by a noconnection DOWN stays retired, exactly as heal_partition/3 leaves it, and a node killed by kill_node/2 stays dead. This restores a network; it does not rewind a run.

-spec heal_partition(dest(), dest()) -> ok.

Heals both directions. See heal_partition/3.

-spec heal_partition(dest(), dest(), event_opts()) -> ok.

Heals both directions, optionally delivering a link-up signal — #{signal => nodeup}, or any of the forms partition/3 takes. See partition/3 for why that matters, and learns for delivering it to one side only.

Resurrects nothing. A monitor that fired noconnection is gone, exactly as it would be in real Erlang, and a system that wants to keep watching its peer has to monitor it again. Healing a link is not undoing the failure; it is a second event.

-spec in_flight() -> non_neg_integer().

How many messages are delayed but not yet delivered.

The network's analogue of a stray timer: non-zero at the end of a run means a message was decided on and never arrived, usually because the run ended or its destination died first.

-spec inherit(pid(), pid()) -> ok.

Places Child where Parent is, with the same faultability. Called by eta_sched as it adopts a process; a harness does not call this. See place/2.

-spec kill_node(term()) -> ok.

Kills a simulated node with no signal. See kill_node/2.

-spec kill_node(term(), event_opts()) -> ok.

"Node N is gone": every process located on it dies, and every survivor gets the events that death produces.

eta_net:kill_node(nb, #{signal => nodedown}).

Five things happen, in this order, and the order is the point — a harness that hand-rolls this from place/2, exit/2 and partition/3 gets it subtly wrong, usually by killing first:

1. Every simulated monitor held from another node on a process that is about to die fires {'DOWN', Ref, process, Object, noconnection}.
2. Every monitor with either end on the node is retired.
3. Messages in flight to or from the node are cancelled, as a failing node loses what was on the wire.
4. Every located process on the node — placed and attached — is killed.
5. Surviving located processes receive {Signal, Node}.

The asymmetry in step 1

Real distribution reports a lost node as noconnection to a remote monitor while a monitor on the same machine sees the true exit reason. One VM cannot do that on its own: a killed process yields killed to every monitor it has. So the remote monitors are retired before the process dies, and the DOWN they receive is the one distribution would have sent. Monitors held by anything not located — a client, a harness, an observer — are untouched real monitors and see killed, which is the same asymmetry seen from the other side.

Step 1 must precede step 4 for a second reason: once the exit signal is out, the scheduler may report it at any moment, and a monitor still on the books then would fire with the real reason.

Atomicity

The whole sequence runs in the driver, between steps, while every process the scheduler owns is suspended. No survivor can observe a half-dead node, because no survivor runs until it is a fully dead one.

Restarting

A killed node's name survives; its processes do not. place/2 or attach/2 on the same name afterwards is how a restart is expressed, and the new processes are ordinary members of that node — they get a fresh position in the link-event fan-out, since the ones they replace are gone.

Two things deliberately do not reset. Any cut involving the node is still in force, so a node that was partitioned and then died comes back partitioned until it is healed; and nothing re-monitors on a survivor's behalf. Both follow from the same rule as everywhere else here: an event says what just happened, it does not undo what happened before.

-spec monitor(process | port | time_offset, term()) -> reference().

erlang:monitor/2, aware of the simulated network.

Most systems detect a failed peer with a monitor rather than with a timeout, so without this a partition is invisible to exactly the code it should be exercising: the target is alive in the same VM, the VM's monitor stays armed, and the link fault drops messages into a system that never learns why.

eta_transform points a module's erlang:monitor/2,3 and demonitor/1,2 here, so ordinary code needs no change.

What is simulated, and what is left to the VM

A monitor is simulated when, at the moment it is created, watcher and target are located on different simulated nodes. Everything else — same node, either end unplaced, no topology at all, a monitor on a port or a time_offset, or one asking for an alias — is a plain erlang:monitor and behaves exactly as it always did.

That line is deliberately narrow, and the reason is which way each choice fails. A simulated monitor is the only kind that can be made to fire noconnection, because the VM will not let one process cancel another's monitor and will not let anyone rewrite a DOWN already on its way — so a monitor that stayed real would report killed to a remote watcher, or report twice. But a simulated monitor depends on this module being told the target died (see notify_exit/2), and if that ever fails to arrive the watcher waits forever. Simulating only the monitors that need it keeps that exposure to the pairs a partition can actually sever, and leaves everything else on machinery that cannot go wrong.

The consequence worth stating: a monitor created before its ends are placed is a real one and will not fire noconnection. Place the topology before the system starts monitoring — init/2, alongside registration — which is where it belongs anyway.

When a simulated monitor fires

Exactly once, then it is gone, as a real one is:

• {'DOWN', Ref, process, Object, noconnection} when partition/3 or kill_node/2 severs the link between watcher and target.
• {'DOWN', Ref, process, Object, Reason} when the target exits, with the real reason, reported by eta_sched. See notify_exit/2.
• {'DOWN', Ref, process, Object, noproc} immediately, if the target is already dead when the monitor is created.
• {'DOWN', Ref, process, Object, noconnection} immediately, if the link between watcher and target is already down when the monitor is created.

Object follows erlang:monitor/2: the pid for a pid, {Name, node()} for a registered name. {tag, Tag} in the options replaces 'DOWN', as it does for the BIF.

The last two are one rule: a monitor that cannot be established says so at once, rather than never. The BIF answers both that way, and code that waits for a DOWN it will not otherwise get depends on it — an agent that monitors a peer to find out it is unreachable waits forever otherwise, and the run looks healthy while it does.

A noconnection DOWN is terminal. Real Erlang does not re-arm a monitor after one, and neither does this: heal_partition/3 resurrects nothing, and a system that wants to keep watching must monitor again. The immediate noconnection above is not an exception to that, because it is not the old failure being replayed. Nothing here is retroactive. It reports the topology now, which is the same thing the BIF reports when it is asked to monitor across a connection it cannot make.

"Already down" means both directions cut at the node level — the state partition/3 leaves behind, and the only one that models a lost connection: real distribution has no one-way link failure, since either end's tick timing out tears the whole connection down. A one-way cut/2, and a cut/2 between two processes, leave the link up by definition; a monitor created across one is established and fires nothing, which is the whole point of that fault. See cut/2.

One approximation, and it is visible only through learns: this reads the topology, not who has noticed it. partition/3 with learns => a leaves side B believing its peer is reachable, and a monitor B creates afterwards still gets the immediate noconnection — the cut is in the table, and B's ignorance of it is not.

-spec node_of(dest()) -> term() | undefined.

The simulated node a process is on, or undefined. See place/2.

-spec notify_exit(pid(), term()) -> ok.

Tells the network a process has exited, so simulated monitors on it can fire.

Called by eta_sched from the exit trace event, not by a harness. Inert when no network is running.

Why the scheduler is the source

A simulated monitor has no VM monitor behind it, so something has to notice the target died. Every candidate but this one is worse:

• A broker process holding real monitors would learn at the right moment and forward at the wrong one — it is not a process the scheduler owns, so its forwarded DOWN lands whenever the BEAM happens to run it. That is a message ordered by wall clock, which is the thing this framework exists to remove.
• A real monitor in the watcher, under a private tag, would land at exactly the right moment in exactly the wrong shape, and no one can rewrite another process's mailbox.
• Polling would fire on whichever driver call happened next.

eta_sched is already the tracer for every process in a run and already calls inherit/2 from the same handler, so it learns of an exit synchronously with the step that caused it, in a process whose mailbox is read at points the schedule fixes. Delivery from there is exact: only one process runs during a step, so a DOWN raised at the end of it sits behind every message the dying process sent during it, which is the order the VM would have produced.

The boundary this leaves: a target the scheduler does not own reports nothing, so a simulated monitor on one never fires. Every process a run schedules is owned by construction — that is processes/1's contract — so this is a statement about processes outside the run, and it is another reason simulation is confined to monitors that cross a declared link.

-spec partition(dest(), dest()) -> ok.

Cuts both directions — which is what a lost link actually is. See partition/3.

-spec partition(dest(), dest(), event_opts()) -> ok.

Cuts both directions, delivers the noconnection DOWNs the failure produces, and optionally a link-down signal.

eta_net:partition(na, nb, #{signal => nodedown}).

The signal

Prefer the signalling form. A real link failure is never only lost messages: it delivers nodedown/nodeup to both ends and systems hang recovery off those. Dropping without signalling injects something no network produces — messages vanishing while both ends still believe the link is up — and the unrecovered state that follows is an artefact rather than a defect.

The opposite trap is real too: recovery driven by the signal may repair the very divergence under test, so a run asserting a property during a fault restricts loss by scope instead. Neither is right in every situation, which is why there is no default.

Three forms, and the first is the one to reach for:

An atom names a kind and is derived per side, because that is what a partition actually says: processes on A learn that B is gone, and processes on B learn that A is gone. One undifferentiated term tells both sides the same thing, which is never what happened. A and B here are the arguments as resolved — a node name if the argument named a node, the pid otherwise.

The literal forms are what a term that is already complete uses, and #{signal => {nodedown, node()}} keeps meaning exactly what it always did. {literal, _} exists for the one case the bare form cannot express: a signal that is a single atom.

Which side learns

learns says which side of the partition observes the failure: both (the default), a, or b. It governs the signal and the DOWNs together, since they are the same event seen twice.

eta_net:partition(na, nb, #{signal => nodedown, learns => a}).

is "A finds out that B is gone, and B does not notice" — an asymmetry real distribution produces constantly, since the two ends time out independently, and one that had to be hand-rolled from two cut/2 calls and a fan-out before. The cut itself stays symmetric: a lost link loses both directions whether or not anyone has realised.

Monitors

Every simulated monitor held across the partition fires exactly one {'DOWN', Ref, process, Object, noconnection} and is retired. See monitor/2 for which monitors those are and why the DOWN is terminal.

Delivery

DOWNs first, then signals; side A's fan-out before side B's; each in placement order. Everything is delivered directly rather than routed, so a cut channel cannot swallow it, and everything is enqueued before this call returns — no process the scheduler owns runs in between, so no one can observe half of it.

-spec place(term(), [dest()]) -> ok.

Puts processes on a simulated node, on the wire — how a system says where its network is.

Faults apply to a send only when both ends are placed and their nodes differ, so the fault model follows the topology rather than a predicate the harness has to keep correct.

eta_net:place(node_a, [MemberA]),
eta_net:place(node_b, [MemberB]).

A placed process is both located and faultable: it receives its node's link events and dies with it (kill_node/2), and its traffic can be dropped, delayed or cut. attach/2 gives the first without the second; see the module doc on why those come apart.

Unplaced means not on the network: never faulted, in either direction, and it receives no link events. That is the safe default and also keeps a network inert for processes it does not know about, such as leftovers from an earlier test in the same VM. It is not the way to say "on this node but not on the wire" — attach/2 is, because an unplaced process is not on any node at all and so learns nothing when one fails.

If nothing at all is placed, every link is faultable, so place/2 is opt-in.

Children inherit their parent's node and its faultability as eta_sched adopts them, so a worker spawned mid-run does not send across a link the network was never told about — nor acquire one its parent was deliberately kept off.

Placing a process that is already placed moves it and keeps its position in the link-event fan-out; see partition/3.

-spec policy() -> policy().

The policy currently in force.

-spec register_name(term(), atom(), dest()) -> ok.

Registers Name on simulated node Node, so {Name, Node} resolves the way {Name, node()} does in a real cluster.

eta_net:place(node_a, [ServerA]),
eta_net:register_name(node_a, my_server, ServerA),
eta_net:cast({my_server, node_a}, hello).   %% reaches ServerA

This is what lets a system keep its own addressing. A distributed Erlang program says {RegisteredName, Node} and lets the VM find the process; there is no way to write that against simulated nodes, because one VM holds one registry and a second gen_server:start_link({local, Name}, ...) fails with already_started. Without this, {Name, SomeSimulatedNode} normalizes to undefined and the send falls through to erlang:send/2, which on a non-distributed VM drops it silently — a whole cluster's traffic vanishing with nothing raised and nothing counted.

The name is scoped to the node, so the same name on two nodes is two processes, which is the point: a per-node singleton is exactly the shape that cannot be expressed otherwise.

Registration is independent of placement — this only answers "who is Name on Node" — but registering without placing is almost always a mistake, since an unplaced process is on no node and is never faulted. Register the pid you placed.

A pid may hold several names, and re-registering a name moves it. Nothing is removed when a process dies: a name whose process is gone resolves to a dead pid, which routes and fails exactly as a stale {Name, Node} does against a real node that has restarted. Use unregister_name/2 to model deregistration.

-spec reply([{reply, term(), term()}] | {reply, term(), term()}) -> ok.

gen_statem:reply/1, routed. Takes one {reply, From, Reply} or a list of them, and puts each on the network through reply/2.

-spec reply(term(), term()) -> ok.

gen_server:reply/2 and gen_statem:reply/2, routed — so a reply can be lost independently of the request that caused it.

Addressed to the caller's pid rather than to the alias in the tag. Both land in the same mailbox and the caller's selective receive matches on the tag either way.

eta_transform also rewrites a gen_server's handle_call/3 returns and a gen_statem's reply actions to reach this, which is how the reply leg of call/3 gets onto the network.

A reply to a caller that already timed out is dropped, which is what the alias gen:call waits on does outside a simulation. See call/3.

-spec running() -> boolean().

Whether a network is running. When false, every function here delegates.

-spec send(dest() | term(), term()) -> term().

Sends a message, subject to the network.

Returns Msg, so it is a drop-in for both Dest ! Msg and erlang:send/2.

A destination this module cannot identify as a local process — {global, _}, {via, _, _}, a remote node — is passed through untouched. eta does not simulate distribution.

-spec send(dest() | term(), term(), list()) -> ok | nosuspend | noconnect.

erlang:send/3. The options are for distribution and have no meaning between two processes in one VM, so a routed send ignores them.

-spec set_policy(policy()) -> ok.

Sets the random fault policy. Anything omitted keeps its current value.

• drop_p — probability that an in-scope message is lost.
• delay_p — probability that it is instead delivered late.
• max_delay — upper bound, in virtual milliseconds, on a delay.
• scope — which traffic the probabilities apply to:
  • all (the default),
  • {tags, [Tag]} — messages carrying one of these tags,
  • fun(From, Dest, Tag) -> boolean().

A message's tag is its leading element, except that cast/2 and reply/2 name the payload rather than OTP's envelope: a cast of {prepare, TxId} is tagged prepare, not '$gen_cast', and every reply is tagged '$gen_reply'.

Scope states which recovery paths a run is entitled to exercise, rather than tuning how hard it tries. Loss on a channel whose recovery depends on a node event cannot be asserted against mid-run, because nothing delivers that event until the heal; a run that checks a property during a fault has to restrict loss to a channel that repairs itself, such as a replication stream a follower rejoins from a version discontinuity.

An out-of-scope message draws nothing from the RNG, so narrowing the scope does not shift the fault schedule for the traffic still inside it.

-spec start() -> ok.

Starts a network with a perfect policy. See start/1.

-spec start(#{seed => integer(), policy => policy()}) -> ok.

Starts the network.

Options:

• seed — seeds the fault schedule, so a run replays.
• policy — see set_policy/1. Defaults to a perfect network, which is what a cluster needs while it is still starting up.

One network per VM, as with eta_time and eta_log, so runs must be serial. Starting over a running network resets it; starting over one another live process owns raises.

-spec statem_return(term()) -> term().

Puts the reply leg of a gen_statem:call on the network, given a state callback's return value.

The gen_server half of this job needs the From out of the callback's arguments, so eta_transform generates a per-module helper for it. A gen_statem names its caller inside the return instead — a {reply, From, Msg} action — so the whole job is a function of the return value alone and lives here.

Every action-carrying return shape is recognised. Each {reply, From, Msg} action is sent through reply/2 and removed from the list, so gen_statem does not also send it directly:

{next_state, S, D, [{reply, From, R}, Other]}  %% -> eta_net:reply(From, R),
                                               %%    {next_state, S, D, [Other]}
{stop_and_reply, Reason, [{reply, From, R}]}   %% -> eta_net:reply(From, R),
                                               %%    {stop_and_reply, Reason, []}

Anything else is returned untouched, including a return with no actions and a return shape this does not know. Idempotent, so a return that has already passed through — from a helper function the transform also wrapped — is unchanged the second time.

Actions may be a single action rather than a list, which is why the bare {reply, _, _} clause exists.

-spec stats() -> #{atom() => non_neg_integer()}.

Message counts by disposition.

dropped is what a non-vacuity guard asserts on: a run whose network never lost anything tested the same system a perfect one would, and reports the same ok.

unmanaged counts sends made by a process other than the one the scheduler was stepping, into a process the scheduler owns — see eta_sched:stepping/0. Non-zero means part of the run was ordered by wall clock, and nothing else reports it. Traffic between two processes the run does not own is delivered untouched and not counted.

signalled and noconnection count link events rather than traffic: node signals delivered, and synthetic noconnection DOWNs fired. They are the non-vacuity guard for a partition, the way dropped is for a lossy policy — a run that partitioned a node nothing was on, or severed no monitor, exercised no recovery and reports the same ok.

-spec stop() -> ok.

Stops the network. Safe to call when none is running.

Messages already in flight are ordinary eta_time timers by then, and are still delivered by whatever advances the clock next.

-spec unregister_name(term(), atom()) -> ok.

Removes a simulated registration. See register_name/3.

-spec unsupported({module(), atom(), arity()}, [term()]) -> term().

What eta_transform points the messaging functions this module does not implement at: broadcasts and multi-node calls, the asynchronous request/response interface, and the gen_event client API. The list is ?NET_UNSUPPORTED in eta_transform.

Raises while a network is running, and calls the original otherwise — so a module holding one on a path no simulation reaches still builds and behaves normally.

Raising rather than passing through is deliberate: a run states a fault model, and a channel the model silently fails to cover makes the suite green for the wrong reason.

-spec whereis_name(term(), atom()) -> pid() | undefined.

The pid registered as Name on simulated node Node, or undefined.

erlang:whereis/1 for the simulated registry. See register_name/3.