Design

How the NIF is put together, for the person about to change it. The user guides say what the library does; this says why it is built the way it is and which rules a change must keep. Every claim here is about the code as it is; when you change the code, change this page in the same commit.

Read this first

The whole runtime rests on one rule:

A Wasmtime store is used by exactly one thread at a time. While the guest runs, that thread is the instance's worker thread. At any other time it is whichever thread holds the instance mutex.

Everything that looks like a lock or a "busy" error is that rule applied. Wasmtime itself has no thread affinity: a store may be driven from any thread, but never from two at once.

Where things are

c_src/nif.h holds every struct and the prototypes shared between files. src/wasmtime.erl is the public API and owns defaults and option validation; src/wasmtime_nif.erl is only the NIF stub table. Options reach the NIF as maps read by key (parse_options, configure_wasi in nif_instantiate.c), never by tuple position, so the two sides cannot drift silently: a missing or ill-typed key is badarg naming the key.

Ownership

flowchart LR
  Erl[Erlang process] -->|holds| H[handle_t]
  Erl -->|holds| M[module resource]
  Erl -->|holds| R[ref_t]
  H -->|1 ref| I[instance_t]
  W[worker thread] -->|1 ref| I
  R -->|1 ref| I
  I -->|1 ref| M
  M -->|uses| E[engine_t, never freed]
  I -->|owns| S[Wasmtime store, linker, hostfns, inbox, capture]
  X[externref payload env] -->|freed by| GC[Wasmtime collector]

Rules that follow:

• A message never carries a resource term. Every message carries the Erlang reference made at instantiate time (#instance.ref, copied into inst->ref_env), so a worker thread cannot resurrect a resource from a mailbox. Refs (ref_t) do travel in messages as results and host call arguments; they are ordinary resources whose lifetime is by refcount.
• No destructor blocks. handle_dtor signals; instance_dtor runs only when the thread is gone; ref_dtor needs no lock.
• The worker thread is detached and owns one instance reference. The handle owns the other. A failed instantiate sets stopping so the thread exits and releases its reference; the handle still exists (the Erlang side got an error and drops it).

The instance state machine

State lives in instance_t under inst->mu; inst->cv is the one condition variable, broadcast whenever something a waiter cares about changed.

Transitions, by who makes them:

"Busy" for scheduler-side store access (with_export, with_ref, with_memory, nif_externref, nif_gc): refused when state == ST_RUNNING. Allowed when idle or ST_IN_HOST, because then the guest is parked and the mutex holder is the only user of the store. A host function therefore may read memory, globals, tables and refs of the instance it runs on, but may not call it (enqueue would queue behind itself).

timeout on call/4 is implemented by the caller: wait_result/3 calls cancel/2 after the timeout; ok means the result will never be sent, not_running means it already is in the mailbox and is returned. settle/2 answers a host call that was in flight so nothing lingers.

Message contracts

Replies go the other way through NIFs, never messages: host_reply/3, send/2, close/1. Terms in a message are built in a fresh enif_alloc_env and sent with enif_send(NULL, ...), the only way a non-scheduler thread may send.

Values

Two paths cross the boundary, chosen per function type by shape_of:

A signature with both v128 and a reference cannot cross (unsupported_type).

Rules:

• Never call wasm_valtype_kind: it aborts on GC and non-nullable types. vtype_of reads wasmtime_valtype_t instead and gives the kind, the family (FAM_NUM, FAM_EXTERN, FAM_FUNC, FAM_ANY, FAM_EXN) and nullability.
• term_to_val always produces an owned root (a clone of the ref_t's root, or a fresh i31). The caller unroots it (unroot_vals) unless the callee takes ownership. wasmtime_func_call does not take ownership of arguments; a typed callback's results are taken over by Wasmtime, and so are its args, which is why host_callback_typed clones before val_to_term.
• val_to_term consumes the value: the resulting ref_t owns the root. It returns 0 for a kind that cannot cross (exnref).
• The kind of a raw value always comes from the function type, never from the value.

Engines and precompiled modules

One engine per distinct compile option set, in a registry keyed by {Fuel, OptLevel, SortedProposalOverrides} (normalised by compile_key/1 on the Erlang side so equal maps mean the same engine), capped at MAX_ENGINES (32) because engines are never freed. Every engine has epoch interruption on and concurrency_support off; the ticker thread bumps all of them every EPOCH_TICK_NS.

A .cwasm (serialize/1, deserialize/1,2) records the engine it was made with. Wasmtime accepts it when:

Consequences: deserialize/1 tries the default engine then the fuel one; deserialize/2 picks by options; a module compiled with proposals off loads anywhere; the full and runtime-only libraries share make_config so they accept the same files; the stdin shim shipped in priv/shims must be compiled with the same settings, which is why scripts/precompile-shims.sh mirrors make_config flag by flag and shim_files_load in the tests fails when they drift.

Streams

Every instance has one inbox (a list of byte chunks under mu). send/2 appends, close/1 marks end of input. Two guest faces read it:

• the erlang.recv import, one whole chunk per call;
• stdin, when stdin => stream: a fd_read defined in the linker in front of WASI's own (wasmtime_linker_allow_shadowing), serving fd 0 from the inbox as a byte stream.

Other fds must still reach Wasmtime's fd_read, but Wasmtime's WASI functions find the guest memory through their caller's memory export, and a host-to-host call has no caller. The shim module (scripts/stdin-shim.wat, embedded as SHIM_WASM) imports the guest memory, exports it as memory and forwards; the override calls the shim's export. A full build compiles the shim once per engine; a runtime-only build has no compiler and loads priv/shims/<platform>-<plain|fuel>.cwasm.

Output is push: the custom stdout/stderr callback (stream_write) and erlang.send deliver one message per write; the mailbox is the buffer. What a write is depends on the guest's C library: it fully buffers a stdout it believes is a file, so a stream stdout or stderr also shadows fd_fdstat_get and reports a character device without seek and tell rights (wasi-libc's isatty test); musl then line-buffers and every line is one message. The shim forwards both fd_read and fd_fdstat_get.

Build pipeline

scripts/fetch-wasmtime.sh resolves the C API in this order: an explicit WASMTIME_C_API_DIR; a cached archive; a download (this repo's release for runtime-only and FreeBSD, upstream otherwise) checked against scripts/*.sha256; a source build. scripts/build-nif.sh probes what the library can do (NIF_HAVE_COMPILER, _WAT, _WASI, by nm or a link test, cross-checked with conf.h), records the platform in priv/wasmtime_platform, and rebuilds only when the stamp file (_build/wasmtime_nif.stamp, the library path and flags) changed.

Decisions and their reasons:

Numbers