Search documentation of macula

# macula_content_chunker (macula v0.44.2)

View Source

Content chunking module for Macula content-addressed storage.

Provides functions to split binary data into fixed-size chunks, reassemble chunks back into the original data, and compute Merkle tree root hashes for verification.

### Chunk Size

The default chunk size is 256KB (262144 bytes), which provides a good balance between: - Network efficiency (not too many small packets) - Memory usage (chunks fit in L2 cache) - Parallelism (enough chunks for multi-provider download)

### Example Usage

       %% Split data into chunks
       {ok, Chunks} = macula_content_chunker:chunk(Data, 262144),

       %% Get chunk metadata with hashes
       Infos = macula_content_chunker:chunk_info(Chunks, sha256),

       %% Compute Merkle root
       Root = macula_content_chunker:merkle_root(Infos, sha256),

       %% Reassemble chunks
       Data = macula_content_chunker:reassemble(Chunks).

# Summary

## Types

chunk_info/0

## Functions

chunk(Data, ChunkSize)

Split binary data into fixed-size chunks. Returns a list of binaries, where all but the last are exactly ChunkSize bytes. The last chunk may be smaller. Empty input returns an empty list.

chunk_info(Chunks, Algorithm)

Create chunk info records with index, offset, size, and hash.

default_chunk_size()

Return the default chunk size (256KB).

merkle_root(Infos, Algorithm)

Compute Merkle tree root hash from chunk infos. Uses a binary tree structure where leaf nodes are chunk hashes and internal nodes are hash(left || right).

reassemble(Chunks)

Reassemble chunks back into original data.

verify_chunk(Chunk, ExpectedHash, Algorithm)

Verify a chunk against its expected hash.

# Types

# chunk_info/0

    -type chunk_info() ::
              #{index := non_neg_integer(),
                offset := non_neg_integer(),
                size := pos_integer(),
                hash := binary()}.

# Functions

# chunk(Data, ChunkSize)

    -spec chunk(binary(), pos_integer()) -> {ok, [binary()]}.

Split binary data into fixed-size chunks. Returns a list of binaries, where all but the last are exactly ChunkSize bytes. The last chunk may be smaller. Empty input returns an empty list.

# chunk_info(Chunks, Algorithm)

    -spec chunk_info([binary()], macula_content_hasher:algorithm()) -> [chunk_info()].

Create chunk info records with index, offset, size, and hash.

# default_chunk_size()

    -spec default_chunk_size() -> pos_integer().

Return the default chunk size (256KB).

# merkle_root(Infos, Algorithm)

    -spec merkle_root([chunk_info()], macula_content_hasher:algorithm()) -> binary().

Compute Merkle tree root hash from chunk infos. Uses a binary tree structure where leaf nodes are chunk hashes and internal nodes are hash(left || right).

# reassemble(Chunks)

    -spec reassemble([binary()]) -> binary().

Reassemble chunks back into original data.

# verify_chunk(Chunk, ExpectedHash, Algorithm)

    -spec verify_chunk(binary(), binary(), macula_content_hasher:algorithm()) -> boolean().

Verify a chunk against its expected hash.