macula_content_fetch (macula v12.10.0)

View Source

Fetches content from the node that shares it (D27): content is available while its sharer is online, and stations only relay.

The fetch

  1. Find the content id's announcements in the DHT (macula_record:content_key/1). Keep those that name the content id, a realm, a serving station and a procedure bound to their announcer: ~<hex>/content_v1 or <org>/content_v1_<hex>, where <hex> is the announcer's node id. An announcement cannot point a fetch at another node's procedure.
  2. Try the sharers one at a time, in a random order.
  3. Resolve the serving station's own endpoint record and open a server_stream through it, pinned to the station and targeted at the sharer, in the announced realm. No advertisement or realm-key check applies: content verifies itself by its content id, so a fetcher that trusts no realm can still fetch.
  4. Ask for the root. A block must hash to the content id; a manifest must match it (macula_manifest:verify_mcid/2) before any of its sizes is read, and then fit the caller's bounds before a chunk is asked for.
  5. Ask for each chunk on its own stream, parallel at a time, each verified against its own content id as it arrives; then assemble in order and verify the whole against the manifest.

A sharer that fails in any step moves the fetch to the next; when every one has failed the answer is {error, {unavailable, [{Sharer, Reason}]}}, naming each. Nothing of a failed attempt is kept, and nothing is resumed. No announcement at all is {error, not_shared}.

Bounds

max_bytes (256 MiB by default) bounds the content, max_chunks (16,384, 4 GiB of 256 KiB chunks) a manifest, chunk_timeout_ms (15 s) each stream, dial and answer together, parallel (4) the chunk streams open at once. A raw root larger than a chunk is refused, and every chunk must be exactly the size its manifest declares, so what a fetch receives never exceeds the manifest's size, which max_bytes bounds.

Processes

The fetch runs in a worker that monitors its caller, and every station resolution and stream ask, the root's and each chunk's, runs in a worker of its own linked to it. One that exits fails its sharer and the fetch moves on; a fetch worker that crashes answers {error, {fetch_worker, Reason}}. Nothing takes the caller down, a gone caller ends the fetch while it waits on any ask, and the caller's mailbox is left as it was.

Summary

Functions

Fetch MCID from a node that shares it. Opts: realm (only that realm's announcements), max_bytes, max_chunks, chunk_timeout_ms, parallel, and io (see io/0).

The facade's own functions, the io a fetch uses when not given another.

Functions

get(Pool, MCID, Opts)

-spec get(pid(), macula:mcid(), map()) -> {ok, binary()} | {error, term()}.

Fetch MCID from a node that shares it. Opts: realm (only that realm's announcements), max_bytes, max_chunks, chunk_timeout_ms, parallel, and io (see io/0).

io()

-spec io() -> map().

The facade's own functions, the io a fetch uses when not given another.