z_url_fetch (zotonic_stdlib v1.35.0)

View Source

Fetch (part of) the data of an Url, including its headers. Also support decoding 'data:' URLs and streaming fetched data to a device. Uses the httpc OTP library for the actual fetching.

The fetch functions are able to unzip partially fetched data.

Set {autoredirect, false} for fixed protocol endpoints: redirect responses are returned as HTTP errors without contacting the Location URL or forwarding credentials.

Cross-origin redirects default to {authorization, none}, which options_fun can override for the destination. Set {authorization_redirect, error} to reject cross-origin redirects with explicit authorization. Same-origin redirects with explicit authorization still require HTTPS.

{authorization, none} suppresses authorization on every request, including redirects.

{options_fun, Fun} resolves request options per destination. Fun(Method, Url, Options) receives the method, normalized binary URL and sanitized caller options (without options_fun). Use it for destination-specific credentials instead of forwarding a fixed token.

Returned options apply only to that request. The original anonymous option always takes precedence. Transfer options (device and max_length) are set by the caller.

Summary

Types

fetch_result/0
option/0
options/0
options_fun/0

Functions

ensure_profiles()
fetch(Url, Options)

Fetch the data and headers from an url

fetch(Method, Url, Payload, Options)

Fetch the data and headers from an url

fetch_partial(Url)

Fetch the first 64 kilobytes of data and headers from an url

fetch_partial(Url, Options)

Fetch the first N bytes of data and headers from an url, optionally save to the file device

fetch_partial(Method, Url, Payload, Options)

Fetch the first N bytes of data and headers from an url, optionally save to the file device

periodic_cleanup()
profile(Url)

Types

fetch_result/0

-type fetch_result() ::
          {ok,
           {FinalUrl :: string(),
            RespHeaders :: [{string(), string()}],
            ContentLength :: non_neg_integer(),
            Content :: binary()}} |
          {error, Reason :: term()}.

option/0

-type option() ::
          {device, pid()} |
          {timeout, pos_integer()} |
          {max_length, pos_integer()} |
          {use_range, boolean()} |
          {autoredirect, boolean()} |
          {insecure, boolean()} |
          {authorization, binary() | string() | none | undefined} |
          {authorization_redirect, none | error} |
          {options_fun, options_fun()} |
          {accept, binary() | string()} |
          {user_agent, binary() | string()} |
          {language, atom()} |
          {content_type, binary() | string()} |
          {headers, [{binary() | string(), binary() | string()}]} |
          insecure.

options/0

-type options() :: [option()].

options_fun/0

-type options_fun() :: fun((get | post | delete | put | patch, binary(), options()) -> options()).

Functions

ensure_profiles()

-spec ensure_profiles() -> ok.

fetch(Url, Options)

-spec fetch(Url, Options) -> fetch_result() when Url :: string() | binary(), Options :: options().

Fetch the data and headers from an url

fetch(Method, Url, Payload, Options)

-spec fetch(Method, Url, Payload, Options) -> fetch_result()
               when
                   Method :: get | post | put | delete | patch,
                   Url :: string() | binary(),
                   Payload :: binary(),
                   Options :: options().

Fetch the data and headers from an url

fetch_partial(Url)

-spec fetch_partial(Url) -> fetch_result() when Url :: string() | binary().

Fetch the first 64 kilobytes of data and headers from an url

fetch_partial(Url, Options)

-spec fetch_partial(Url, Options) -> fetch_result()
                       when Url :: string() | binary(), Options :: options().

Fetch the first N bytes of data and headers from an url, optionally save to the file device

fetch_partial(Method, Url, Payload, Options)

-spec fetch_partial(Method, Url, Payload, Options) -> fetch_result()
                       when
                           Method :: get | post | delete | put | patch,
                           Url :: string() | binary(),
                           Payload :: binary(),
                           Options :: options().

Fetch the first N bytes of data and headers from an url, optionally save to the file device

periodic_cleanup()

-spec periodic_cleanup() -> ok.

profile(Url)

-spec profile(string() | binary()) -> atom().