ImagePipe.Source.HTTP (image_pipe v0.5.0)

Copy Markdown View Source

Source adapter for images fetched over HTTP and HTTPS.

web: [
  adapter: ImagePipe.Source.HTTP,
  match: [scheme: ["http", "https"]],
  options: [allowed_hosts: ["assets.example.com"]]
]

It resolves URL sources such as https://assets.example.com/beach.jpg and, with :base_url, path sources. Setting up a mount is covered in Serving images from an HTTP origin.

Before connecting, and again for each redirect, the adapter checks the scheme and host, resolves the host, and denies the fetch unless every resolved address is public or allowed by :address_policy. It then connects to a checked address while keeping the hostname for the Host header and TLS verification. The default resolver keeps an answer for 10 seconds, so a busy origin isn't looked up for every fetch (see :dns_cache). The addresses are still checked on every fetch. When none of them accepts a connection or answers in time, the adapter looks the host up again and tries a changed answer once. Why is explained in Source network policy. A denied fetch fails with {:source, :denied_scheme}, {:source, :denied_host}, or {:source, :denied_address}, and a host that doesn't resolve fails with {:source, :unresolved_host}. All of them answer 404.

Options

  • :allowed_hosts (list of String.t/0) - Hosts the adapter may connect to, compared without case. A bare entry such as "assets.example.com" allows only the scheme's default port, 80 or 443. Name another port as "assets.example.com:8443", or "[::1]:8080" for an IPv6 address. Redirects can't leave the list. Required unless :base_url is set, which defaults it to the base URL's host and port. A list given with :base_url must allow that host and port.

  • :base_url (String.t/0) - Origin that path sources are served from, such as "https://images.example.com/originals". Each path segment is percent-encoded and appended, so beach.jpg fetches https://images.example.com/originals/beach.jpg. Must be an http or https URL with a host and no query, fragment, or credentials. Without it, the adapter serves only URL sources.

  • :path_pattern (Regex.t/0) - Regular expression that the whole path, segments joined with /, must match. Other paths are not found, and the origin is never contacted. Requires :base_url.

  • :req_options (keyword/0) - Options for Req.request/1, such as headers: or auth:. The adapter drops the :url, :base_url, :method, :body, :params, :into, :retry, :redirect, and :max_redirects options, the host header, and the conditional headers if-none-match, if-modified-since, if-match, if-unmodified-since, and if-range. Setting Req's :adapter or :unix_socket is a configuration error, because both connect somewhere other than the address the source checked (Source network policy). It also drops range, accept, and accept-encoding headers, and the :range option, except on a source with internal_cache: :disabled that isn't immutable. Requests for one URL must always return the same bytes, because cached originals and processed images are reused per URL. The default value is [].

  • :receive_timeout (non_neg_integer/0) - Milliseconds to wait for the response and between body chunks. The default value is 5000.

  • :fetch_timeout (pos_integer/0) - Milliseconds for the whole download, from the first host name lookup to the last body byte, including redirects. A connection attempt can run past it by up to connect_timeout. Requests waiting for another request's download or check of the same original stop waiting after this long plus one second. The default value is 30000.

  • :connect_timeout (non_neg_integer/0) - Milliseconds to wait for a connection. The default value is 5000.

  • :pool_timeout (non_neg_integer/0) - Milliseconds to wait for a free connection from the pool. A request that waits longer fails with 504. The default value is 5000.

  • :pool_size (pos_integer/0) - Maximum number of open HTTP/1.1 connections to each address the origin's host resolves to. Each connection carries one download at a time, so further downloads from the same address wait for a free connection, up to pool_timeout. The default value is 50.

  • :max_redirects (non_neg_integer/0) - Redirects to follow. Each target is checked against :allowed_hosts and :address_policy. A redirect to another host, port, or scheme is sent without the :req_options headers and credentials. A redirect beyond the limit, from https to http, or to a URL with a user name or password fails the request with 502. The default value is 0.

  • :address_policy (keyword/0 or (:inet.ip_address(), atom() -> boolean())) - Which non-public addresses the adapter may connect to. See Address policy. The default value is [].

  • :address_resolver ((String.t() -> {:ok, [:inet.ip_address()]} | {:error, term()})) - Resolves a hostname to the addresses to check and connect to, such as a caching resolver. An address the adapter connected to for the same scheme, host, and port in about the last minute is tried first, because it may still have an open connection. The rest are tried in the order returned, without duplicates. An error, an empty list, or an exception fails the fetch with {:source, :unresolved_host}. When none of the addresses accepts a connection or answers in time, the resolver is called once more, and a changed answer is tried. The default resolver returns IPv4 addresses before IPv6 addresses. A resolver passed here replaces it, so :dns_cache doesn't apply.

  • :dns_cache (boolean/0) - Whether the default resolver keeps the addresses a hostname resolves to for 10 seconds. Turn it off when the server's DNS resolver already caches and its TTLs should apply exactly. The default value is true.

  • :stable (:auto or :immutable) - :immutable marks an immutable source: an identifier always names the same bytes, so its originals and processed images never expire and are never revalidated. With :auto, each original is identified by its content. See immutable sources. The default value is :auto.

  • :cache_policy (keyword/0) - Cache policy for this source, with the fields of ImagePipe.Source.CachePolicy. Each field it sets replaces that field of the :source_cache_policy of ImagePipe.config/1. A source with stable: :immutable can't set a :freshness duration or a :stale_while_revalidate window. The default value is [].

  • :internal_cache (:enabled or :disabled) - :disabled keeps this source's originals and processed images out of the caches, including images that use this source for a watermark. :enabled caches what the source's storage policy allows. The default value is :enabled.

  • :http_cache (:inherit, :validators, :auto, :public, or :private) - HTTP cache headers for responses from this source. :inherit uses the mount's :http_cache (see ImagePipe.Plug), and any other value replaces it. See per-source modes. The default value is :inherit.

Address policy

By default the adapter connects only to public addresses. :address_policy takes a keyword list that also allows some categories or ranges:

address_policy: [allow: ["10.0.5.0/24"], allow_loopback: true]
  • :allow - a list of CIDR ranges, such as "10.0.5.0/24" or "fd00::/8".
  • :allow_loopback - 127.0.0.0/8 and ::1.
  • :allow_unspecified - 0.0.0.0/8 and ::.
  • :allow_link_local - 169.254.0.0/16 and fe80::/10.
  • :allow_private - 10.0.0.0/8, 172.16.0.0/12, and 192.168.0.0/16.
  • :allow_unique_local - fc00::/7.
  • :allow_multicast - 224.0.0.0/4 and ff00::/8.
  • :allow_broadcast - 255.255.255.255.
  • :allow_cgnat - 100.64.0.0/10.
  • :allow_reserved - other non-public ranges: benchmarking and documentation ranges, IETF protocol assignments (192.0.0.0/24 and 2001::/23, which includes Teredo), the 6to4 relay 192.88.99.0/24, 240.0.0.0/4, and IPv6 addresses outside 2000::/3.

Each category takes true or false. Categories treat IPv4-mapped, NAT64 (64:ff9b::/96), and 6to4 IPv6 addresses as the IPv4 address they contain. An IPv4 range in :allow also matches an address in IPv4-mapped or NAT64 form, such as ::ffff:10.0.5.7 or 64:ff9b::10.0.5.7, but not in 6to4 form.

Or pass a function that receives each resolved address as a tuple and its category (:public, or one of the categories above without allow_) and returns true to allow it. It replaces the built-in decision, and any result other than true, or an exception, denies the address:

address_policy: fn _ip, category -> category in [:public, :private] end

HTTP/2

The adapter fetches over HTTP/1.1 by default. It reuses idle connections, but each connection carries one fetch at a time, so concurrent fetches open a connection each. protocols: [:http2] in :req_options fetches over HTTP/2 from an origin that supports it, such as Cloudflare or Google Cloud Storage:

options: [
  allowed_hosts: ["assets.example.com"],
  req_options: [connect_options: [protocols: [:http2]]]
]

Concurrent fetches from that origin then share one connection per resolved address. Against an origin that only speaks HTTP/1.1, such as an Amazon S3 endpoint, every fetch fails with {:source, :connect_error}.

protocols: [:http2, :http1] works with any origin and uses HTTP/2 when the origin offers it, but still sends one request at a time on each connection.