wasm_code_cache (wasm v0.3.0)

View Source

Compiled code, kept on disk so a node restart does not pay for it again.

Compiling QuickJS is about twenty seconds of a core. Nothing waits for it, so it is not a latency problem, but it is twenty seconds every time a node starts and every time an instance of a module it has never seen goes hot. This makes it once.

Off unless you turn it on. Reading a .beam from disk and loading it is executing whatever is in that file, so the directory holding the cache is as trusted as the code in your release. Wasmtime's cache is opt-in for the same reason. Set it in the application environment:

application:set_env(wasm, code_cache_dir, "/var/cache/my_app/wasm").

What is checked, and what is only claimed

"As trusted as your release" was once only written down. It is checked now, and the difference is worth being precise about, because the checks are narrower than they look.

A path is refused unless it is absolute with no dot component, the directory is owned by this node's user with no group or other write bit, every directory above it is owned by root or that user and equally unwritable by others, and nothing on the path is a symlink. An entry is refused unless it is a regular file whose framed digest matches. Every refusal is a miss and a single line in the log; nothing here raises, because a cache that cannot be trusted is a slower node and never a broken one.

Ancestors are checked for ownership and not only for mode, which is the part that is easy to leave out: a directory owned by somebody else at 0755 is not group-writable, and its owner can still rename or replace everything beneath it.

What none of it does: the digest detects damage -- a torn write, a bad disk, a crash -- and not somebody who can write a well-formed entry. Validating a path and opening a file under it are not atomic either. Neither gap is closed, and under this model neither needs to be: once every ancestor is owned by root or the node's user and writable by nobody else, only those two can change what the path resolves to, and both can already run code in the node. These checks catch misconfiguration, not an attacker who is already inside. Trusting artifacts from a party that may not run code here would need them authenticated, which this does not do.

Why the uid comes from a probe

Nothing in Erlang answers "what uid is this node". file:read_file_info/1 takes an open descriptor as well as a name, so the answer comes from a file this process is holding rather than from whatever a name refers to by the time a stat runs: open a probe exclusive, stat the descriptor, close and delete. Statting a path instead would be a race, and a private directory to hold it would be a more elaborate way of not needing one.

What a key covers

Everything that would make an artifact wrong if it changed, which is more than the module:

  • the module's content hash, so two different modules never collide
  • the ABI between generated code and wasm_exec
  • the OTP release and the emulator flavour, because generated BEAM is only loadable by the emulator that compiled it
  • the machine's architecture
  • the compiler quality asked for, since baseline and full are different code
  • the set of functions compiled, because the tier compiles what ran and two workloads reach different sets
  • the slot the artifact was built for, because a module's name is part of its BEAM file and cannot be changed without rewriting it

A module identified by a reference() rather than a content hash is never cached. That is every module built from text: its identity is fresh on every validation, so there is nothing stable to key on.

What it does not do

No sharing between nodes, no signature, no compression. A cache entry is a file named for the hash of its key -- framed with a magic, a format number, a length and a digest, in the shape wasm_snapshot_file uses -- and eviction is by total size, oldest first.

The rename that publishes an entry and the digest inside it do different jobs. The rename stops a reader seeing a half-written file; it does not give durability across a crash, which would need an fsync nothing here does. The digest is what turns damage from a crash into a miss.

Summary

Functions

Where the cache lives, or undefined when it is off.

The key for one artifact, or undefined when this module cannot be cached.

The artifact for this key, if there is one and it is readable.

Throw the cache away. For tests, and for a release that wants a clean start.

Keep this artifact under this key.

Functions

dir()

-spec dir() -> undefined | file:filename().

Where the cache lives, or undefined when it is off.

key/6

-spec key(term(), non_neg_integer(), module(), baseline | full, [non_neg_integer()], term()) ->
             binary() | undefined.

The key for one artifact, or undefined when this module cannot be cached.

Identity is #module.identity: only the {sha256, _} form is stable enough to key on, and a reference() answers undefined here rather than being hashed, because a fresh reference every validation would fill the cache with entries nothing can ever hit.

lookup(Key)

-spec lookup(binary()) -> {ok, binary()} | miss.

The artifact for this key, if there is one and it is readable.

Any failure is a miss. A cache that cannot be read is a slower start and never an error, which is the same rule the rest of the tier follows: every refusal means do the work.

purge()

-spec purge() -> ok.

Throw the cache away. For tests, and for a release that wants a clean start.

store(Key, Bin)

-spec store(binary(), binary()) -> ok.

Keep this artifact under this key.

Written to a temporary name and renamed, because a half-written .beam that a later start reads is a crash rather than a miss, and rename is atomic on every filesystem this runs on.