Manifest read/write for cross-VM incremental compilation.
Serializes database state (memo entries, entity tables, intern tables,
revision counters) and source file metadata to disk. On the next
mix compile, the manifest is loaded to restore the database to its
previous state, enabling incremental batch compilation without a
long-lived VM.
What gets persisted
- Input memo entries (all durabilities) — needed so unchanged files can be skipped entirely on warm start.
- Derived memo entries with durability
:highor:medium(:lowderived entries like hover info are cheap to recompute). - Entity table data (identity keys, tracked fields, refcounts).
- Intern table data (the forward mapping, encoded, and counter state;
the reverse mapping is rebuilt when the table is first used — see
Roux.Intern.encode_snapshot/1). - Revision counter and durability tracking state.
- Source file metadata (mtime, content hash) for staleness detection.
Layout (format 5)
A manifest is a header and a payload:
<<"ROUXMNFT", format::32, crc32(payload)::32, payload::binary>>The payload is one uncompressed term_to_binary/1 of the manifest
data. What makes it fast to load is what that term holds: each memo
entry's value is already a binary of its own (the external term
format, compressed at level 1 — Roux.Memo.persisted/2), and so is
each intern table's forward rows (Roux.Intern.encode_snapshot/1).
Decoding the payload decodes keys, dependencies and revisions, and
copies those binaries without looking inside them. restore/2 inserts
the memo entries with their values still encoded and leaves the intern
rows pending: a value is decoded by the first read that needs it, and
an intern table loads on its first miss. A warm run reads a handful of
values and no interned symbol, so it pays for almost none of it; on a
350-module scry project, loading and restoring the manifest went from
300 ms to under 20.
Writing is the same in reverse. A value restored from the last manifest and never replaced goes back out in the encoding it came in with, and an intern table nothing used hands back its encoded rows, so a run only encodes what it recomputed.
Each entry also carries its code version (Roux.Query) and the blob
digests its value names (Roux.Runtime.hold/1).
Values held by digest
With a Roux.Blob store (the database's, Roux.Database.new/1's
blob:, or write/4's), the value of a store: :blob query is kept
in the store and the manifest holds its digest: a large value the
manifest need not carry, read back by the first read that needs it.
Its early cutoff compares digests, and a value whose blob is gone is
recomputed transparently (Roux.Runtime). The manifest is the owner
of what it names (Roux.Blob.retain/3): every held digest and every
one an entry holds, so a collection of the store keeps them for as
long as the manifest is there.
Integrity
The header's CRC-32 covers the payload. load/1 checks it before
decoding the payload, and the decoded term's shape before returning
it, so a truncated or corrupted file is refused as a whole, never
partly read; the values decoded later are bytes the checksum covered. write/3 writes a
temporary file beside the manifest and renames it over the old one:
a reader sees the old manifest or the new one, and a write that dies
halfway leaves the old one in place. On some file systems (APFS) that
replacing rename leaves the name missing for a moment, so load/1
reads again, twice, before it takes a missing manifest for none: a
run that did meet the moment starts cold, which costs time and never
a wrong result.
Versioning
The format number in the header changes whenever the layout does; a
manifest of any other format — including formats 1 to 3, which were a
bare term_to_binary/2 of the data with the version inside, and 4,
whose entries had no code version or blobs — is refused, and the
caller rebuilds from scratch.
Summary
Types
Deserialized manifest data. The memo entries' values and the intern
tables' rows are still encoded; see memo_entries/1.
Metadata for a single source file.
Functions
Loads a manifest from disk.
The memo entries a loaded manifest carries, decoded: [{query_key, entry}],
values held by digest read from store (:missing without it, or when
their blob is gone).
Restores a database from manifest data.
Collects mtime and content hash for a list of source file paths.
Writes a manifest to disk, atomically: the file at path is the old
manifest or the new one, never part of either.
Types
@type manifest_data() :: %{ vsn: pos_integer(), sources: %{required(String.t()) => source_meta()}, memo_entries: [Roux.Memo.persisted()], entity_data: [{module(), list()}], intern_data: [{atom(), Roux.Intern.encoded_snapshot()}], revision: map() }
Deserialized manifest data. The memo entries' values and the intern
tables' rows are still encoded; see memo_entries/1.
Metadata for a single source file.
Functions
@spec load(String.t()) :: {:ok, manifest_data()} | :error
Loads a manifest from disk.
Returns {:ok, data} for a manifest of this format whose checksum and
shape hold, :error otherwise (missing file, another format, a
truncated or corrupted file).
@spec memo_entries(manifest_data(), Roux.Blob.t() | nil) :: [ {Roux.Memo.query_key(), Roux.Memo.Entry.t() | :missing} ]
The memo entries a loaded manifest carries, decoded: [{query_key, entry}],
values held by digest read from store (:missing without it, or when
their blob is gone).
restore/2 never decodes the values — each is decoded by the first
read that needs it — so this is for inspection.
@spec restore(Roux.Database.t(), manifest_data()) :: :ok
Restores a database from manifest data.
Populates the revision counter, memo table, entity tables, and intern
tables from the serialized state, leaving memo values and intern rows
encoded until they are first used (see "Layout"). The database should
be freshly created (via Database.new/0), with its queries registered,
before calling this:
- an entry of a query that is not registered is left out: nothing could re-execute it, or tell which code computed it — and so is every entry that read one, directly or through others, whether or not the manifest kept the unregistered query's own entry. Kept, it would hold an edge nothing can bring up to date, and its readers' durability checks would pass over it;
- an entry computed by another code version than its query's
(
Roux.Query) is restored, and stale: it re-executes when next demanded, keeping itschanged_atif its value comes back the same. The revision advances at:highonce when there is any, since no durability check sees a code change.
A value held by digest is read from the database's Roux.Blob store.
@spec source_metadata([String.t()]) :: %{required(String.t()) => source_meta()}
Collects mtime and content hash for a list of source file paths.
@spec write( Roux.Database.t(), %{required(String.t()) => source_meta()}, String.t(), keyword() ) :: :ok
Writes a manifest to disk, atomically: the file at path is the old
manifest or the new one, never part of either.
Values and intern rows restored from the last manifest and not replaced since are written in the encoding they came in with.
An entry whose query says so is left out (Roux.Query's store: :none), as is a transient entry (transient:) and every entry that
read one, directly or through others.
Options
:blob— theRoux.Blobstore to keepstore: :blobvalues in (default: the database's). Without one they are written inline. The manifest then retains what it names in the store (see "Values held by digest").