Roux.Lang.Manifest (roux v0.3.2)

Copy Markdown View Source

Manifest read/write for cross-VM incremental compilation.

Serializes database state (memo entries, entity tables, intern tables, revision counters) and source file metadata to disk. On the next mix compile, the manifest is loaded to restore the database to its previous state, enabling incremental batch compilation without a long-lived VM.

What gets persisted

  • Input memo entries (all durabilities) — needed so unchanged files can be skipped entirely on warm start.
  • Derived memo entries with durability :high or :medium (:low derived entries like hover info are cheap to recompute).
  • Entity table data (identity keys, tracked fields, refcounts).
  • Intern table data (the forward mapping, encoded, and counter state; the reverse mapping is rebuilt when the table is first used — see Roux.Intern.encode_snapshot/1).
  • Revision counter and durability tracking state.
  • Source file metadata (mtime, content hash) for staleness detection.

Layout (format 8)

A manifest is a header and a payload:

<<"ROUXMNFT", format::32, crc32(payload)::32, payload::binary>>

The payload is an uncompressed ETF term. Keys and dependencies share dictionary IDs, and repeated paths and digests share binary references. Loading expands these references before validating the memo rows. Memo values remain separate ETF binaries, compressed at level 1 (Roux.Memo.persisted/2). Each intern table's forward rows are also encoded separately (Roux.Intern.encode_snapshot/1).

restore/2 inserts memo entries with their values still encoded and leaves intern rows pending. Values decode on their first read; intern tables load on their first miss. A warm run pays only for the values and tables it uses.

Writing is the same in reverse. A value restored from the last manifest and never replaced goes back out in the encoding it came in with, and an intern table nothing used hands back its encoded rows, so a run only encodes what it recomputed.

Each entry also carries its code version (Roux.Query) and the blob digests its value names (Roux.Runtime.hold/1).

Values held by digest

With a Roux.Blob store (the database's, Roux.Database.new/1's blob:, or write/4's), the value of a store: :blob query is kept in the store and the manifest holds its location: a large value the manifest need not carry, read back by the first read that needs it. Its early cutoff compares digests, and a value whose blob is gone is recomputed transparently (Roux.Runtime). The manifest retains physical value files and every artifact digest recorded by Roux.Runtime.hold/1, so the store keeps them while the manifest exists. A record's logical digest compares values; its pack's digest is the garbage-collection root.

Integrity

The header's CRC-32 covers the payload. load/1 checks it before decoding the payload, and the decoded term's shape before returning it, so a truncated or corrupted file is refused as a whole, never partly read; the values decoded later are bytes the checksum covered. write/3 writes a temporary file beside the manifest and renames it over the old one: a reader sees the old manifest or the new one, and a write that dies halfway leaves the old one in place. On some file systems (APFS) that replacing rename leaves the name missing for a moment, so load/1 reads again, twice, before it takes a missing manifest for none: a run that did meet the moment starts cold, which costs time and never a wrong result.

Versioning

Format 8 shares keys, dependencies, code versions and binary subterms. Formats 5, 6 and 7 remain readable; older releases reject format 8 and rebuild. Format 7 added packed memo-value locations. Each packed value keeps its logical digest independently of its physical file and byte range. Unchanged locations are reused; sparse packs are compacted within a bounded write budget. Format 6 added code-version observations to dependency lists.

Summary

Types

manifest_data()

Deserialized manifest data. The memo entries' values and the intern tables' rows are still encoded; see memo_entries/1.

source_meta()

Metadata for a single source file.

Functions

load(path)

Loads a manifest from disk.

memo_entries(map, store \\ nil)

The memo entries a loaded manifest carries, decoded: [{query_key, entry}], values held by digest read from store (:missing without it, or when their blob is gone).

restore(db, data)

Restores a database from manifest data.

source_metadata(paths)

Collects mtime and content hash for a list of source file paths.

write(db, source_metadata, path, opts \\ [])

Writes a manifest to disk, atomically: the file at path is the old manifest or the new one, never part of either.

Types

manifest_data()

@type manifest_data() :: %{
  vsn: pos_integer(),
  sources: %{required(String.t()) => source_meta()},
  memo_entries: [Roux.Memo.persisted()],
  entity_data: [{module(), list()}],
  intern_data: [{atom(), Roux.Intern.encoded_snapshot()}],
  revision: map()
}

Deserialized manifest data. The memo entries' values and the intern tables' rows are still encoded; see memo_entries/1.

source_meta()

@type source_meta() :: %{mtime: term(), hash: integer()}

Metadata for a single source file.

Functions

load(path)

@spec load(String.t()) :: {:ok, manifest_data()} | :error

Loads a manifest from disk.

Returns {:ok, data} for a manifest of this format whose checksum and shape hold, :error otherwise (missing file, another format, a truncated or corrupted file).

memo_entries(map, store \\ nil)

@spec memo_entries(manifest_data(), Roux.Blob.t() | nil) :: [
  {Roux.Memo.query_key(), Roux.Memo.Entry.t() | :missing}
]

The memo entries a loaded manifest carries, decoded: [{query_key, entry}], values held by digest read from store (:missing without it, or when their blob is gone).

restore/2 never decodes the values — each is decoded by the first read that needs it — so this is for inspection.

restore(db, data)

@spec restore(Roux.Database.t(), manifest_data()) :: :ok

Restores a database from manifest data.

Populates the revision counter, memo table, entity tables, and intern tables from the serialized state, leaving memo values and intern rows encoded until they are first used (see "Layout"). The database should be freshly created (via Database.new/0), with its queries registered, before calling this:

  • an entry of a query that is not registered is left out: nothing could re-execute it, or tell which code computed it — and so is every entry that read one, directly or through others, whether or not the manifest kept the unregistered query's own entry. Kept, it would hold an edge nothing can bring up to date, and its readers' durability checks would pass over it;
  • an entry computed by another code version than its query's (Roux.Query) is restored, and stale: it re-executes when next demanded, keeping its changed_at if its value comes back the same. The revision advances at :high once when there is any, since no durability check sees a code change.

A value held by digest is read from the database's Roux.Blob store.

source_metadata(paths)

@spec source_metadata([String.t()]) :: %{required(String.t()) => source_meta()}

Collects mtime and content hash for a list of source file paths.

write(db, source_metadata, path, opts \\ [])

@spec write(
  Roux.Database.t(),
  %{required(String.t()) => source_meta()},
  String.t(),
  keyword()
) :: :ok

Writes a manifest to disk, atomically: the file at path is the old manifest or the new one, never part of either.

Values and intern rows restored from the last manifest and not replaced since are written in the encoding they came in with.

An entry whose query says so is left out (Roux.Query's store: :none), as is a transient entry (transient:) and every entry that read one, directly or through others. With reverse tracking, entries without a valid dependency proof and their readers are also omitted.

Options

  • :blob — the Roux.Blob store to keep store: :blob values in (default: the database's). Without one they are written inline. The manifest then retains what it names in the store (see "Values held by digest").