internetdata_http (internetdata v1.0.0)
View SourceThe transport: one httpc profile, retry policy, and response classification.
A request either buffers its body or streams it into a SINK. A catalog listing is small enough to hold; a dataset file is not, and the published catalog runs to gigabytes, so the sink is a fold that never sees more than one chunk at a time.
Summary
Functions
Start what the default transport needs.
Fetch and decode a JSON body, retrying what is worth retrying.
Fetch the Location of a redirect without following it.
Stream one absolute URL through Sink, answering what it wrote.
The default transport, closing over nothing.
Types
-type request() :: #{method := get, url := binary(), headers := [{binary(), binary()}], timeout_ms := pos_integer(), sink => sink()}.
-type result() :: #{headers := [{binary(), binary()}], status => 100..599, body => binary(), written => non_neg_integer(), acc => term()}.
Functions
-spec ensure_ready() -> ok.
Start what the default transport needs.
-spec get_json(map(), binary(), [{binary(), binary()}], non_neg_integer()) -> {ok, map()} | {error, internetdata_error:error()}.
Fetch and decode a JSON body, retrying what is worth retrying.
-spec get_redirect(map(), binary(), [{binary(), binary()}], non_neg_integer()) -> {ok, binary()} | {error, internetdata_error:error()}.
Fetch the Location of a redirect without following it.
-spec get_stream(map(), binary(), sink(), non_neg_integer()) -> {ok, #{written := non_neg_integer(), acc := term()}} | {error, internetdata_error:error()}.
Stream one absolute URL through Sink, answering what it wrote.
No credential is attached. This is only ever pointed at the presigned link the download endpoint hands out, which authorizes itself, and forwarding the API key would hand it to a host with no business holding it.
-spec httpc_fun() -> http_fun().
The default transport, closing over nothing.