internetdata_http (internetdata v1.6.0)

View Source

The transport: one httpc profile, retry policy, and response classification.

A request either buffers its body or streams it into a SINK. A catalog listing is small enough to hold; a dataset file is not, and the published catalog runs to gigabytes, so the sink is a fold that never sees more than one chunk at a time.

Summary

Types

fold/0
http_fun/0
request/0
response/0
result/0
sink/0

Functions

ensure_ready()

Start what the default transport needs.

escape(Value)
get_json(Client, Path, Query, Retries)

Fetch and decode a JSON body, retrying what is worth retrying.

get_redirect(Client, Path, Query, Retries)

Fetch the Location of a redirect without following it.

get_stream(Client, Url, Sink, Retries)

Stream one absolute URL through Sink, answering what it wrote.

httpc_fun()

The default transport, closing over nothing.

oauth(Client, Request, Retries, Decode)

Send one OAuth request, attempted again up to Retries times. A 2xx goes to Decode with its status; a 4xx naming an RFC 6749 error is that refusal, which is never retryable; anything else is the ordinary error for its status.

oauth_request(_, Path, Form)

A request to the authorization server, carrying NO credential whatever the client holds: these endpoints have no use for the API key, and on the token endpoint an authorization header reads as client authentication. A form is POSTed, its type riding as the content-type header whole/1 hands to httpc.

Types

fold/0

-type fold() :: fun((binary(), term()) -> {ok, term()} | {error, term()}).

http_fun/0

-type http_fun() :: fun((request()) -> response()).

request/0

-type request() ::
          #{method := get | post,
            url := binary(),
            headers := [{binary(), binary()}],
            timeout_ms := pos_integer(),
            body => binary(),
            sink => sink()}.

response/0

-type response() :: {ok, result()} | {error, term()}.

result/0

-type result() ::
          #{headers := [{binary(), binary()}],
            status => 100..599,
            body => binary(),
            written => non_neg_integer(),
            acc => term()}.

sink/0

-type sink() :: #{fold := fold(), acc := term()}.

Functions

ensure_ready()

-spec ensure_ready() -> ok.

Start what the default transport needs.

escape(Value)

get_json(Client, Path, Query, Retries)

-spec get_json(map(), binary(), [{binary(), binary()}], non_neg_integer()) ->
                  {ok, map()} | {error, internetdata_error:error()}.

Fetch and decode a JSON body, retrying what is worth retrying.

get_redirect(Client, Path, Query, Retries)

-spec get_redirect(map(), binary(), [{binary(), binary()}], non_neg_integer()) ->
                      {ok, binary()} | {error, internetdata_error:error()}.

Fetch the Location of a redirect without following it.

get_stream(Client, Url, Sink, Retries)

-spec get_stream(map(), binary(), sink(), non_neg_integer()) ->
                    {ok, #{written := non_neg_integer(), acc := term()}} |
                    {error, internetdata_error:error()}.

Stream one absolute URL through Sink, answering what it wrote.

No credential is attached. This is only ever pointed at the presigned link the download endpoint hands out, which authorizes itself, and forwarding the API key would hand it to a host with no business holding it.

httpc_fun()

-spec httpc_fun() -> http_fun().

The default transport, closing over nothing.

oauth(Client, Request, Retries, Decode)

-spec oauth(map(),
            request(),
            non_neg_integer(),
            fun((100..599, binary()) -> {ok, term()} | {error, internetdata_error:error()})) ->
               {ok, term()} | {error, internetdata_error:error()}.

Send one OAuth request, attempted again up to Retries times. A 2xx goes to Decode with its status; a 4xx naming an RFC 6749 error is that refusal, which is never retryable; anything else is the ordinary error for its status.

oauth_request(_, Path, Form)

-spec oauth_request(map(), binary(), [{binary(), binary()}] | undefined) -> request().

A request to the authorization server, carrying NO credential whatever the client holds: these endpoints have no use for the API key, and on the token endpoint an authorization header reads as client authentication. A form is POSTed, its type riding as the content-type header whole/1 hands to httpc.