Macula SDK — Content Guide

Content-addressed blob storage on the mesh, by MCID.

Audience: applications that store and fetch immutable blobs — files, snapshots, artifacts — and want integrity for free. Single-block storage since SDK 4.2.7; chunked content and discovery since 8.10.0; direct-dial fetch and seed (get_content_station/put_content_station, macula_download/macula_feeder's start_link_direct) since 9.6.0/9.7.0. For a live, open-ended feed instead of a fixed blob, see the Streaming Guide.

Overview

Macula content is content-addressed: a blob is named by the hash of its bytes, not by where it lives. That name is an MCID (Macula Content ID). Two consequences follow directly, and they are the whole point:

• Integrity is self-verifying. The name is the hash (or, for larger content, a Merkle root over chunk hashes), so a fetched blob can be checked against its own MCID — a corrupted or substituted blob fails the check by construction.
• Location stops mattering. Any host holding the bytes serves the same MCID, so hosts are interchangeable and content deduplicates naturally.

{ok, MCID}  = macula:put_content(Pool, Bytes),
{ok, Bytes} = macula:get_content(Pool, MCID).

put_content/2 stores Bytes and returns its MCID — transparently as one block for small content, or as a chunked, Merkle-verified manifest for large content (see below). get_content/2 fetches the bytes back for either shape, or {error, not_found} if no reachable host holds a copy. Integrity is verified before the bytes are returned, so the caller does not re-verify.

MCID format

An MCID is a 34-byte binary:

<<Version:8, Codec:8, Hash:32/binary>>
    1         1        32

• byte 0 — version (1)
• byte 1 — codec: 16#55 (raw) for a single block, 16#56 (manifest) for chunked content
• bytes 2..33 — the 32-byte hash: BLAKE3 of the bytes (raw), or BLAKE3 over a canonical encoding of the manifest's metadata (manifest — see below)

Because the MCID is derived purely from the content, storing identical bytes always yields the same MCID — that is what makes it a content address.

Supervised wrappers: macula_feeder / macula_download

put_content/2 and get_content/2 are plain blocking calls — no addressable pid to cancel one from outside. macula_feeder and macula_download wrap them as proper OTP behaviours: start_link/4,5 returns immediately with a pid, runs the put/get in a linked worker, and delivers the outcome to your callback. Each publishes its own sharing.put_started_v1 / sharing.put_completed_v1 (feeder) or sharing.get_started_v1 / sharing.get_completed_v1 (download) mesh fact, carrying chunked => true | false so an observer can tell single-block transfers from manifest ones without decoding the MCID itself.

-module(doc_download).
-behaviour(macula_download).
-export([init/1, handle_downloaded/2]).

init(Parent) -> {ok, Parent}.

handle_downloaded(Result, Parent) ->
    Parent ! {downloaded, Result},
    {stop, normal, Parent}.

{ok, Pid} = macula_download:start_link(doc_download, Pool, Realm,
                                       Mcid, self()).

%% cancel before the get resolves — publishes sharing.get_completed_v1
%% with outcome => cancelled
ok = macula_download:cancel(Pid).

macula_feeder is the symmetric put-side counterpart (Module:handle_fed/2 in place of handle_downloaded/2). Embed macula_feeder_sup / macula_download_sup (each a simple_one_for_one factory) in your own supervision tree to enumerate or cancel in-flight transfers via supervisor:which_children/1 / terminate_child/2.

Direct-dial: start_link_direct

macula_download:start_link_direct/4,5 resolves a chunked MCID's provider from its content_announcement and dials that station directly — no change needed on the macula_feeder side, since content_announcement is published automatically (see Discovery below). macula_feeder:start_link_direct/5,6 is the put-side counterpart: a PUT has no discovery step, so it takes the target Station (a pubkey) directly instead of resolving one — deliberately seeding that specific station rather than whichever the pool would pick.

{ok, Pid} = macula_download:start_link_direct(doc_download, Pool, Realm,
                                              Mcid, self()),

{ok, FeedPid} = macula_feeder:start_link_direct(doc_feeder, Pool, StationPubkey,
                                                Realm, Bytes, self()).

Only chunked content is discoverable this way — a single-block put is never announced (see Discovery below), so start_link_direct has nothing to resolve for one.

Single block vs. chunked

put_content/2 picks the shape for you, by size, against macula_manifest:default_chunk_size/0 (256 KiB):

The single-block shape is unchanged since v4.2.7 — same MCID formula (<<1, 16#55, BLAKE3(Bytes)>>), same single RPC round trip. It is not a special case bolted on top of chunking; a one-chunk manifest's chunk MCID is identical to the single-block MCID, so the two shapes agree at the boundary.

For content over the chunk size, put_content/2:

1. splits Bytes into fixed-size chunks (macula_manifest:create/1);
2. uploads each chunk via _content.put_block (BLAKE3-verified by the station, same as single-block);
3. builds a manifest — chunk count, per-chunk offsets/sizes/hashes, and a Merkle root over the chunk hashes — and uploads it via _content.put_manifest;
4. returns the manifest's own MCID (codec 16#56).

get_content/2 on a manifest MCID fetches the manifest, then every chunk in order, reassembles, and verifies the whole against the manifest's size and Merkle root before returning — a tampered or truncated chunk is caught before the caller ever sees the bytes.

A chunk failure during put stops immediately without uploading the manifest — a manifest naming missing chunks would resolve but never reassemble, which is worse than a clean error.

The whole transfer — every _content.put_block/_content.get_block call plus the manifest call for chunked content — rides one dedicated QUIC stream, opened once and reused for the sequence, isolating a large blob transfer from other RPC/PubSub traffic on the same connection.

Discovery: who has this MCID?

Chunked content gets announced automatically: when a station stores a manifest, it publishes a signed content_announcement DHT record naming itself as a host, the same way a station announces its endpoint. Resolve every host currently announcing an MCID:

{ok, Providers} = macula:find_content_providers(Pool, MCID),
%% [#{announcer_node := StationPubkey, endpoint := <<"quic://host:443">>,
%%    name := ..., size := ..., chunk_count := ...}, ...]

Each entry's record signature is verified, AND its signer checked to be exactly the announcer_node it claims — not just any valid signature — before its endpoint is trusted; unverifiable, signer-mismatched, or malformed records are dropped silently, never surfaced as an error. Single-block content is not announced (there is no manifest-stored event to trigger it) — resolving its MCID returns {ok, []}, not an error.

Dialing a specific host directly

get_content/2 already reaches a copy via the connected station's own 1-hop peer relay — for most topologies that is enough. When it is not (a partial-mesh pair with no mutual peer, or you want to route around a specific host deliberately), dial an announced host directly with get_content_station/4,5 — the content-transfer counterpart to direct-dial RPC's call_station/6,7:

{ok, [#{announcer_node := Node, endpoint := Url} | _]} =
    macula:find_content_providers(Pool, MCID),
{ok, Bytes} = macula:get_content_station(Pool, Url, MCID, 30_000,
                                         #{expected_node_id => Node,
                                           pin_tls_cert => false,
                                           verify => none}).

Or resolve and fetch in one call with macula_direct_dial:get_content/3, which does exactly the above — retrying past DHT propagation lag the same way find_content_providers/2 returning an empty list means "not yet replicated," not "doesn't exist":

{ok, Bytes} = macula_direct_dial:get_content(Pool, MCID, 30_000).

pin_tls_cert => false matters against a real production station: its TLS is terminated by an unrelated PKI (Let's Encrypt), so pinning the cert's own key can never succeed there — trust instead rests on the application-layer CONNECT/HELLO handshake (see the RPC Guide for the full mechanism). Unlike RPC, content direct-dial has no cert-chain-equivalent opt-in: content is content-addressed and independently re-hashed client-side regardless of which peer serves it, so a rogue or unauthorized announcer can at most refuse to serve or waste a dial — never make a caller accept content that doesn't hash to the MCID it asked for.

Seeding a specific station directly — the put-side counterpart, with no discovery step since you already know which station you're choosing — uses put_content_station/4,5 or macula_direct_dial:put_content/4, resolving Station's own station_endpoint the same way RPC resolves a serving_station:

{ok, MCID} = macula_direct_dial:put_content(Pool, StationPubkey, Bytes, 30_000).

Guarantees reach in one hop regardless of the connected station's relay hop budget — the same value call_station already gives unary RPC calls.

When to use content vs. records vs. streaming

Content is for bytes addressed by what they are; records are for signed statements addressed by who said them; streaming is for a flow with no end known in advance.

Reference

_content.* CALLs retry on a BOLT#4-retryable error (e.g. temporary_relay_failure) up to 3 attempts with a short backoff, per that error's own documented retry contract.