macula_content_fetch (macula v12.10.0)
View SourceFetches content from the node that shares it (D27): content is available while its sharer is online, and stations only relay.
The fetch
- Find the content id's announcements in the DHT (
macula_record:content_key/1). Keep those that name the content id, a realm, a serving station and a procedure bound to their announcer:~<hex>/content_v1or<org>/content_v1_<hex>, where<hex>is the announcer's node id. An announcement cannot point a fetch at another node's procedure. - Try the sharers one at a time, in a random order.
- Resolve the serving station's own endpoint record and open a
server_streamthrough it, pinned to the station and targeted at the sharer, in the announced realm. No advertisement or realm-key check applies: content verifies itself by its content id, so a fetcher that trusts no realm can still fetch. - Ask for the root. A block must hash to the content id; a manifest must match it (
macula_manifest:verify_mcid/2) before any of its sizes is read, and then fit the caller's bounds before a chunk is asked for. - Ask for each chunk on its own stream,
parallelat a time, each verified against its own content id as it arrives; then assemble in order and verify the whole against the manifest.
A sharer that fails in any step moves the fetch to the next; when every one has failed the answer is {error, {unavailable, [{Sharer, Reason}]}}, naming each. Nothing of a failed attempt is kept, and nothing is resumed. No announcement at all is {error, not_shared}.
Bounds
max_bytes (256 MiB by default) bounds the content, max_chunks (16,384, 4 GiB of 256 KiB chunks) a manifest, chunk_timeout_ms (15 s) each stream, dial and answer together, parallel (4) the chunk streams open at once. A raw root larger than a chunk is refused, and every chunk must be exactly the size its manifest declares, so what a fetch receives never exceeds the manifest's size, which max_bytes bounds.
Processes
The fetch runs in a worker that monitors its caller, and every station resolution and stream ask, the root's and each chunk's, runs in a worker of its own linked to it. One that exits fails its sharer and the fetch moves on; a fetch worker that crashes answers {error, {fetch_worker, Reason}}. Nothing takes the caller down, a gone caller ends the fetch while it waits on any ask, and the caller's mailbox is left as it was.
Summary
Functions
-spec get(pid(), macula:mcid(), map()) -> {ok, binary()} | {error, term()}.
Fetch MCID from a node that shares it. Opts: realm (only that realm's announcements), max_bytes, max_chunks, chunk_timeout_ms, parallel, and io (see io/0).
-spec io() -> map().
The facade's own functions, the io a fetch uses when not given another.