wasm_code_slots (wasm v0.1.0)
View SourceWho is still using a piece of generated code, and when its slot may be reused.
You get here if you turn a WebAssembly module into a BEAM module. Read this before you load one, because loading generated code is the part of a compiler that goes wrong quietly: the code runs, the numbers look good, and the failure arrives later as a process killed mid-call or a call that resolved to the wrong module.
What it is
A fixed pool of slots. Each slot owns one pre-interned module name,
wasm_code_0 through wasm_code_N, written into this module's source and
interned when it is compiled. Nothing derived from a user's module ever becomes
an atom, which is the property that matters: the atom table is node-wide and
never reclaimed, so a generated name taken from a module's own bytes is a
permanent leak with a remote attacker holding the tap.
A slot is claimed for a content key -- whatever the caller uses to identify a module, normally its hash -- and is held by leases. There are two kinds and both are load-bearing:
- an instance lease, taken when an instance that may run this code exists
- a call lease, taken for the duration of a call into it
An instance can be destroyed while a call into it is still running, so releasing the instance's lease must not let the code be replaced underneath the call. Reuse happens only when the last lease of either kind is gone.
The two are taken by different mechanisms, and that is a measurement rather
than an inconsistency. An instance lease is rare and is a gen_server call,
about 5.8 us, which buys a monitor so that a dead holder gives its lease back.
A call lease is on the call path -- an interpreted call costs 44 ns -- so it is
one atomics increment on a per-slot counter, about 20 ns, the same shape
wasm_heap uses for execution leases.
When to use it
case wasm_code_slots:claim_loading(Key, {instance, Ref}, self()) of
{compile, Mod, Token} ->
case load(Mod, generate()) of
ok -> wasm_code_slots:publish(Token);
error -> wasm_code_slots:abort(Token)
end;
{resident, _Mod} -> ok;
loading -> interpret;
{error, no_slot} -> interpret
end,
...
case wasm_code_slots:lease_call(Mod, Key) of
ok -> try Mod:F(Args) after wasm_code_slots:release_call(Mod) end;
stale -> interpret
end,
...
ok = wasm_code_slots:release(Key, {instance, Ref}).Nothing enters a slot between the reservation and publish/1. The counter
stays at its exclusive value for the whole reservation, which is what stops a
second caller running the slot's previous occupant while the new binary is
still being compiled.
claim_loading/3 answers {error, no_slot} when every slot is held, and
loading when somebody else is already filling this key in. Neither is a
failure, both are the signal to interpret this module instead. A compiler
that cannot fall back has to either kill a caller or grow the atom table, and
both are worse than being slower.
What it deliberately does not do
It never calls code:purge/1, which kills processes still running old code.
Reuse goes through code:soft_purge/1, and a slot whose old code is still
running is left alone and reported as unavailable rather than taken.
soft_purge/1 is the authority and the call leases are a hint. It was the
other way round once and could not be: a lease is given back in an after,
which does not run when a process is killed untrappably, and killing a process
is how a runaway invocation is stopped here. Leases leak, so they cannot be what
decides whether a slot may be taken.
What makes reuse safe is not here at all. Generated code carries the stamp it
was built for and refuses a caller carrying another, which is atomic with the
call in a way no lease can be. See wasm_core:module/6. That is why a stuck
counter can be repaired from soft_purge/1 without a race.
soft_purge/1 runs inside this server, so every claim, publish and lease
queues behind an operation whose cost grows with the number of live processes on
the node. It is a known serialisation point, kept deliberately: moving it out
re-opens the window the reservation exists to close, and nothing has measured it
as a problem.
It does not survive a process killed outright. exit(Pid, kill) skips the
after that would give a call lease back, and that slot's counter stays raised
for ever, so it is never reused again. The consequence is that callers
interpret instead, which is the fallback this module exists to provide, so this
costs speed and never safety. wasm_heap accepts the same exposure on the same
reasoning.
Summary
Functions
Give up a reservation. The slot goes back to the pool.
How many calls are inside a slot's code. For tests.
Reserve a slot for Key, held by Lease on behalf of Owner.
Create the slot table. Called by the supervisor, before the manager.
Count one call on a module, and say whether it has reached After.
Another lease on a slot already claimed for Key.
Take a call lease on an already-resolved slot.
Take a call lease on a slot, by the module name claim/3 answered with.
Take a call lease and check that the slot still holds the code you meant.
The counter and index a slot's call lease uses, resolved once.
The module Key is loaded into, without taking a lease.
Finish a reservation, after code:load_binary/3 has succeeded.
Give up one lease. The slot becomes reusable when the last one goes, and not before: an instance lease released while a call is still running leaves the code exactly where it is.
Give back a lease taken with lease_at/2.
Give back a call lease.
{Name, Key, LeaseCount} for every slot that holds something.
The module Key is resident in, as a select rather than a listing.
The module name in one slot, by index.
Every slot name, in order. The complete set of atoms this module makes.
Types
Functions
-spec abort(token()) -> ok.
Give up a reservation. The slot goes back to the pool.
-spec calls_in(module()) -> non_neg_integer().
How many calls are inside a slot's code. For tests.
-spec claim_loading(key(), lease(), pid()) -> {compile, module(), token()} | {resident, module()} | loading | {error, no_slot}.
Reserve a slot for Key, held by Lease on behalf of Owner.
Four answers, and they are four situations rather than two:
{compile, Name, Token}— the slot is yours, empty, and held exclusively until youpublish/1orabort/1. Nothing can enter it meanwhile.{resident, Name}— already loaded. A lease was added; nothing was reloaded.loading— somebody else is filling this key in. Interpret. Waiting on another process's compilation is a latency hazard and interpreting is always correct.{error, no_slot}— every slot is held. Interpret.
Owner is monitored for the duration of the reservation as well as for the
lease, so a compiler that dies does not hold the slot for ever.
-spec ensure_table() -> ok.
Create the slot table. Called by the supervisor, before the manager.
-spec hot(key(), pos_integer()) -> boolean().
Count one call on a module, and say whether it has reached After.
Counts per module rather than per instance: a workload of short-lived instances
would otherwise never get hot. The row is dropped the moment the threshold is
reached, so this costs an ets:update_counter/3 during warmup and nothing at
all afterwards, and the table does not grow with every module ever seen.
Another lease on a slot already claimed for Key.
-spec lease_at(atomics:atomics_ref(), pos_integer()) -> ok | stale.
Take a call lease on an already-resolved slot.
The same counter lease_call/2 takes, without the row lookup that checks which
key the slot holds. That check is for a caller which remembered a slot and was
overtaken by a reuse, and it is not the only thing standing in the way: generated
code refuses a caller whose stamp is not the one it was built for. A caller whose
stamp is the module's own content hash therefore does not need the row, because
another module's code cannot match that hash. A caller identified by a
reference() has no stamp but the slot generation, which only the row can give,
and keeps using lease_call/2.
-spec lease_call(module()) -> ok | stale.
Take a call lease on a slot, by the module name claim/3 answered with.
This is the one on the call path, so it is an atomics increment and not a
message: about 20 ns against the 5.8 us a manager round trip costs. It names
the slot rather than the key, because "do not replace this code while I am
inside it" is a property of the slot.
Answers stale rather than waiting when the slot is held exclusively, which is
a reservation in progress. Interpret in that case.
Pair it with release_call/1 under try ... after, so a throw still gives it
back. A process killed outright leaves the count raised and that slot is then
never reused, which costs speed and never safety: a caller who cannot get a
slot interprets instead, which is what this module is built to do.
-spec lease_call(module(), key()) -> {ok, non_neg_integer()} | stale.
Take a call lease and check that the slot still holds the code you meant.
The counter protects the slot; it says nothing about which module is in it. A caller that remembered a name can be overtaken by a reuse between remembering and leasing, and would then run somebody else's code. This takes the lease first, so nothing can change underneath the check, and gives it back if the answer is no.
-spec lease_ref(module()) -> {atomics:atomics_ref(), pos_integer()}.
The counter and index a slot's call lease uses, resolved once.
lease_call/2 looks both up on every call, and so does release_call/1: a
persistent_term:get and a name-to-index dispatch, four times across a call
that may be a few hundred nanoseconds. They are fixed for the life of a slot
name, so a caller that will enter the same code repeatedly resolves them once
and uses lease_at/2 and release_at/2.
The module Key is loaded into, without taking a lease.
Reading without claiming is the point: wasm_jit:await/2 uses it to tell a
module that is still compiling from one that is ready, and claiming there would
take the free slot away from the compiler that is filling it.
-spec publish(token()) -> ok | stale.
Finish a reservation, after code:load_binary/3 has succeeded.
Answers stale when the slot has moved on to a later reservation, which is what
the generation in the token is for: a compiler slow enough to be overtaken must
not publish over whatever took its place.
Give up one lease. The slot becomes reusable when the last one goes, and not before: an instance lease released while a call is still running leaves the code exactly where it is.
-spec release_at(atomics:atomics_ref(), pos_integer()) -> ok.
Give back a lease taken with lease_at/2.
-spec release_call(module()) -> ok.
Give back a call lease.
-spec resident() -> [{module(), key(), non_neg_integer()}].
{Name, Key, LeaseCount} for every slot that holds something.
The module Key is resident in, as a select rather than a listing.
On the path every call takes when a module is hot and not yet adopted, so it is a match spec the emulator runs over sixteen rows and not a term built for the caller to filter. Asking the gen_server instead cost 320 microseconds a call on a three-microsecond one.
-spec slot_module(pos_integer()) -> module().
The module name in one slot, by index.
lists:nth/2 over the name list is fine where a slot is resolved once per
invocation and not where it is resolved per call, which is what re-entry from
the interpreter made it. A tuple index instead, and the tuple is a literal.
-spec slots() -> [module()].
Every slot name, in order. The complete set of atoms this module makes.