ImagePipe.Source.HTTP (image_pipe v0.1.0)

Copy Markdown View Source

Source adapter for images fetched over HTTP and HTTPS.

web: [
  adapter: ImagePipe.Source.HTTP,
  match: [scheme: ["http", "https"]],
  options: [allowed_hosts: ["assets.example.com"]]
]

It resolves URL sources such as https://assets.example.com/beach.jpg and, with :base_url, path sources. Setting up a mount is covered in Serving images from an HTTP origin.

Before connecting, and again for each redirect, the adapter checks the scheme and host, resolves the host, and denies the fetch unless every resolved address is public or allowed by :address_policy. It then connects to a checked address while keeping the hostname for the Host header and TLS verification. Why is explained in Source network policy. A denied fetch fails with {:source, :denied_scheme}, {:source, :denied_host}, or {:source, :denied_address}, which answer 404.

Options

  • :allowed_hosts (list of String.t/0) - Hostnames the adapter may connect to, compared without case. Redirects can't leave the list. Required unless :base_url is set, which defaults it to the base URL's host. A list given with :base_url must include that host.

  • :base_url (String.t/0) - Origin that path sources are served from, such as "https://images.example.com/originals". Each path segment is percent-encoded and appended, so beach.jpg fetches https://images.example.com/originals/beach.jpg. Must be an http or https URL with a host and no query, fragment, or credentials. Without it, the adapter serves only URL sources.

  • :path_pattern (Regex.t/0) - Regular expression that the whole path, segments joined with /, must match. Other paths are not found, and the origin is never contacted. Requires :base_url.

  • :req_options (keyword/0) - Options for Req.request/1, such as headers: or auth:. The adapter drops :url, :base_url, :method, :body, :params, :into, :retry, :redirect, and :max_redirects, and a host header. Setting Req's :adapter or :unix_socket is a configuration error, because both connect somewhere other than the address the source checked (Source network policy). It also drops range, accept, and accept-encoding headers, except on a source with internal_cache: :disabled that isn't immutable. Requests for one URL must always return the same bytes, because cached originals and processed images are reused per URL. The default value is [].

  • :receive_timeout (non_neg_integer/0) - Milliseconds to wait for the response and between body chunks. The default value is 5000.

  • :fetch_timeout (pos_integer/0) - Milliseconds for the whole download, from the first request to the last body byte, including redirects. A connection attempt can run past it by up to connect_timeout. Requests waiting for the same original give up after waiting this long plus one second. The default value is 30000.

  • :connect_timeout (non_neg_integer/0) - Milliseconds to wait for a connection. The default value is 5000.

  • :pool_timeout (non_neg_integer/0) - Milliseconds to wait for a free connection from the pool. The default value is 5000.

  • :max_redirects (non_neg_integer/0) - Redirects to follow. Each target is checked against :allowed_hosts and :address_policy. A redirect beyond the limit fails the request with 502. The default value is 0.

  • :address_policy (keyword/0 or (:inet.ip_address(), atom() -> boolean())) - Which non-public addresses the adapter may connect to. See Address policy. The default value is [].

  • :address_resolver ((String.t() -> {:ok, [:inet.ip_address()]} | {:error, term()})) - Resolves a hostname to the addresses to check and connect to, such as a caching resolver. An address the adapter connected to for the same scheme, host, and port in about the last minute is tried first, because it may still have an open connection. The rest are tried in the order returned, without duplicates. An error, an empty list, or an exception denies the fetch. The default resolver returns IPv4 addresses before IPv6 addresses.

  • :stable (:auto or :immutable) - :immutable marks an immutable source: an identifier always names the same bytes, so its originals and processed images never expire and are never revalidated. With :auto, each original is identified by its content. See immutable sources. The default value is :auto.

  • :cache_policy (keyword/0) - Cache policy for this source, with the fields of ImagePipe.Source.CachePolicy. Each field it sets replaces that field of the :source_cache_policy of ImagePipe.config/1. A source with stable: :immutable can't set a :freshness duration or a :stale_while_revalidate window. The default value is [].

  • :internal_cache (:auto, :enabled, or :disabled) - :disabled keeps this source's originals and processed images out of the caches, including images that use this source for a watermark. :auto and :enabled cache what the source's storage policy allows. The default value is :auto.

  • :http_cache (:inherit, :validators, :auto, :public, or :private) - HTTP cache headers for responses from this source. :inherit uses the mount's :http_cache (see ImagePipe.Plug), and any other value replaces it. See per-source modes. The default value is :inherit.

Address policy

By default the adapter connects only to public addresses. :address_policy takes a keyword list that also allows some categories or ranges:

address_policy: [allow: ["10.0.5.0/24"], allow_loopback: true]
  • :allow - a list of CIDR ranges, such as "10.0.5.0/24" or "fd00::/8".
  • :allow_loopback - 127.0.0.0/8 and ::1.
  • :allow_unspecified - 0.0.0.0/8 and ::.
  • :allow_link_local - 169.254.0.0/16 and fe80::/10.
  • :allow_private - 10.0.0.0/8, 172.16.0.0/12, and 192.168.0.0/16.
  • :allow_unique_local - fc00::/7.
  • :allow_multicast - 224.0.0.0/4 and ff00::/8.
  • :allow_broadcast - 255.255.255.255.
  • :allow_cgnat - 100.64.0.0/10.
  • :allow_reserved - other non-public ranges: benchmarking and documentation ranges, 240.0.0.0/4, NAT64 64:ff9b::/96, and IPv6 addresses outside 2000::/3.

Each category takes true or false. Categories treat IPv4-mapped and 6to4 IPv6 addresses as the IPv4 address they contain. An IPv4 range in :allow also matches an address in IPv4-mapped form, such as ::ffff:10.0.5.7, but not in 6to4 form.

Or pass a function that receives each resolved address as a tuple and its category (:public, or one of the categories above without allow_) and returns true to allow it. It replaces the built-in decision, and any result other than true, or an exception, denies the address:

address_policy: fn _ip, category -> category in [:public, :private] end

HTTP/2

The adapter fetches over HTTP/1.1 by default. It reuses idle connections, but each connection carries one fetch at a time, so concurrent fetches open a connection each. protocols: [:http2] in :req_options fetches over HTTP/2 from an origin that supports it, such as Cloudflare or Google Cloud Storage:

options: [
  allowed_hosts: ["assets.example.com"],
  req_options: [connect_options: [protocols: [:http2]]]
]

Concurrent fetches from that origin then share one connection per resolved address. Against an origin that only speaks HTTP/1.1, such as an Amazon S3 endpoint, every fetch fails with {:source, :connect_error}.

protocols: [:http2, :http1] works with any origin and uses HTTP/2 when the origin offers it, but still sends one request at a time on each connection.