macula_content_fetch (macula v13.3.0)

View Source

Fetches content from the node that shares it (D27): content is available while its sharer is online, and stations only relay.

The fetch

  1. Find the content id's announcements in the DHT (macula_record:content_key/1). Keep those that name the content id, a realm, a serving station and a procedure bound to their announcer: ~<hex>/content_v1 or <org>/content_v1_<hex>, where <hex> is the announcer's node id. An announcement cannot point a fetch at another node's procedure.
  2. Try the sharers one at a time, in a random order.
  3. Resolve the serving station's own endpoint record and open a server_stream through it, pinned to the station and targeted at the sharer, in the announced realm. No advertisement or realm-key check applies: content verifies itself by its content id, so a fetcher that trusts no realm can still fetch. With no signed advertisement there is no KEM key to seal to either, so the stream is opened in the clear, and says so (confidential => off, E2E design §8.1): a relaying station can read the transfer of content that anyone holding its content id may fetch anyway.
  4. Ask for the root. A block must hash to the content id; a manifest must match it (macula_manifest:verify_mcid/2) before any of its sizes is read, and then fit the caller's bounds before a chunk is asked for.
  5. Ask for each chunk on its own stream, parallel at a time, each verified against its own content id as it arrives; then assemble in order and verify the whole against the manifest.

A sharer that fails in any step moves the fetch to the next; when every one has failed the answer is {error, {unavailable, [{Sharer, Reason}]}}, naming each. Nothing of a failed attempt is kept, and nothing is resumed. No announcement at all is {error, not_shared}.

Bounds

max_bytes (256 MiB by default) bounds the content, max_chunks (16,384, 4 GiB of 256 KiB chunks) a manifest, chunk_timeout_ms (15 s) each stream, dial and answer together, parallel (4) the chunk streams open at once. A raw root larger than a chunk is refused, and every chunk must be exactly the size its manifest declares, so what a fetch receives never exceeds the manifest's size, which max_bytes bounds.

Processes

The fetch runs in a worker that monitors its caller, and every station resolution and stream ask, the root's and each chunk's, runs in a worker of its own linked to it. One that exits fails its sharer and the fetch moves on; a fetch worker that crashes answers {error, {fetch_worker, Reason}}. Nothing takes the caller down, a gone caller ends the fetch while it waits on any ask, and the caller's mailbox is left as it was.

Summary

Functions

Fetch MCID from a node that shares it. Opts: realm (only that realm's announcements), max_bytes, max_chunks, chunk_timeout_ms, parallel, and io (see io/0).

The facade's own functions, the io a fetch uses when not given another.

Functions

get(Pool, MCID, Opts)

-spec get(pid(), macula:mcid(), map()) -> {ok, binary()} | {error, term()}.

Fetch MCID from a node that shares it. Opts: realm (only that realm's announcements), max_bytes, max_chunks, chunk_timeout_ms, parallel, and io (see io/0).

io()

-spec io() -> map().

The facade's own functions, the io a fetch uses when not given another.