macula_manifest (macula v10.24.0)

Fixed-size chunking, Merkle-root computation, and manifest construction for content larger than one storage block.

Mirrors macula-station's macula_manifest / macula_content_chunker / macula_content_hasher BYTE-FOR-BYTE: same MCID format, same default chunk size (256 KiB), same Merkle fold, same canonical-CBOR MCID derivation, same manifest wire shape. This is deliberate, not incidental — the SDK puts a manifest via the station's existing (unmodified) _content.put_manifest / _content.get_manifest RPCs, so the two sides must agree on the algorithm bit-for-bit. Both use the same BLAKE3 NIF (macula_blake3_nif, SDK-owned; the station calls it too) and the same deterministic CBOR encoder (macula_record_cbor, SDK-owned; the station's manifest module calls it directly), so this is a faithful client-side port, not a re-derivation.

MCID format (34 bytes): <<Version:8, Codec:8, Hash:32/binary>>. ?CODEC_RAW (16#55) addresses a single chunk (or a whole blob that fits in one chunk — see the module doc on macula:put_content/2). ?CODEC_MANIFEST (16#56) addresses a manifest describing many chunks.

Summary

Types

algorithm/0

chunk_info/0

manifest/0

mcid/0

Functions

chunk_mcid(_, Index, Algorithm)

The MCID a chunk at Index is stored/fetched under. The station derives this same value independently when serving the chunk, so both sides agree on its address without exchanging it.

create(Data)

create(Data, Opts)

Split Data into fixed-size chunks and build its manifest. Returns the manifest and the chunk bytes in order (index 0 first), so a caller can upload each chunk (via _content.put_block) and then the manifest (via _content.put_manifest). Options

default_chunk_size()

from_wire(M)

Read a manifest as it arrives over _content.get_manifest: the station stores + returns the map exactly as its RPC layer decoded it, with no dedicated re-encode/decode round trip on either side — so the shape depends on the general CALL-result codec, not the canonical {text,_} record shape. Robust to atom keys, to binary-string keys (mirroring macula_record:payload_field/2), and to {text, Bin} keys: the frame decoder resolves a key to an atom only when that atom already exists, so in a node that has not yet loaded this module the field names arrive as text. A name or hash algorithm sent as text is read as its binary value, and a missing hash algorithm is blake3. A manifest without an mcid, whose chunks are not a list of maps, or whose hash algorithm is not one this module knows, is {error, invalid_manifest}.

verify(Manifest, Data)

Verify reassembled Data against Manifest: size, then a fresh Merkle root over Data re-chunked the same way.

verify_mcid(Manifest, Mcid)

Check that Manifest describes the content Mcid names: the MCID recomputed from the manifest's canonical fields (name, size, chunk_size, chunk_count, hash_algorithm and root_hash) must equal Mcid. The manifest's own mcid field is not consulted; a sender can put anything there. A manifest missing a canonical field, whose name is not valid UTF-8 text, or whose hash algorithm is not one this module knows, does not describe Mcid either.

Types

algorithm/0

-type algorithm() :: blake3 | sha256.

chunk_info/0

-type chunk_info() ::
          #{index := non_neg_integer(),
            offset := non_neg_integer(),
            size := pos_integer(),
            hash := binary()}.

manifest/0

-type manifest() ::
          #{mcid := mcid(),
            version := pos_integer(),
            name := binary(),
            size := non_neg_integer(),
            created := non_neg_integer(),
            chunk_size := pos_integer(),
            chunk_count := non_neg_integer(),
            hash_algorithm := algorithm(),
            root_hash := binary(),
            chunks := [chunk_info()]}.

mcid/0

-type mcid() :: <<_:272>>.

Functions

chunk_mcid(_, Index, Algorithm)

-spec chunk_mcid(manifest(), non_neg_integer(), algorithm()) -> {ok, mcid()} | {error, invalid_index}.

The MCID a chunk at Index is stored/fetched under. The station derives this same value independently when serving the chunk, so both sides agree on its address without exchanging it.

Algorithm is accepted but unused: the chunk's hash was already computed (with whichever algorithm create/2 was given) and stored in Chunks, so there's nothing left to derive here. KNOWN GAP (2026-09-05, will not be fixed without a design decision): per-chunk fetch verification (macula_content_transfer:verify_block_hash/2) is hardcoded to blake3 regardless of this field, so a manifest created with hash_algorithm => sha256 produces chunk MCIDs that can never actually verify on fetch. hash_algorithm today only affects the manifest's own root-hash Merkle step (verify/2). Same gap independently confirmed in macula-io/macula-rust's content.rs (block_mcid is likewise hardcoded to blake3). Left as-is: nothing exercises sha256 in practice, and whether per-chunk sha256 verification was ever meant to work is an open question, not a bug with an obvious fix.

create(Data)

-spec create(binary()) -> {ok, manifest(), [binary()]}.

create(Data, Opts)

-spec create(binary(), map()) -> {ok, manifest(), [binary()]}.

name — content name (default <<"unnamed">>)
chunk_size — bytes per chunk (default default_chunk_size/0)
hash_algorithm — blake3 | sha256 (default blake3)

default_chunk_size()

-spec default_chunk_size() -> pos_integer().

from_wire(M)

-spec from_wire(map()) -> {ok, manifest()} | {error, invalid_manifest}.

verify(Manifest, Data)

-spec verify(manifest(), binary()) -> ok | {error, size_mismatch | root_hash_mismatch}.

Verify reassembled Data against Manifest: size, then a fresh Merkle root over Data re-chunked the same way.

verify_mcid(Manifest, Mcid)

-spec verify_mcid(manifest(), mcid()) -> ok | {error, manifest_mcid_mismatch}.