Search documentation of Localize

# Localize.Utils.Http (Localize v0.39.0)

Copy Markdown View Source

Supports securely downloading HTTPS content.

This module provides HTTP GET functionality using the built-in `:httpc` client with certificate verification enabled by default. It follows the erlef security guidelines for secure TLS connections.

The primary public API consists of:

- `get/2` - download content from a URL, returning the body on success.

- `get_with_headers/2` - download content from a URL, returning both headers and body on success.

- `certificate_locations/0` - return the list of possible certificate store locations.

## Summary

### Functions

ca_trust_option()

Resolve the trust source to hand to `:ssl` for peer verification.

certificate_locations()

Return all possible locations of a certificate file.

dynamic_certificate_locations()

Return the dynamically discovered certificate file locations.

get(url, options \\ [])

Securely download HTTPS content from a URL.

get_with_headers(request, options \\ [])

Securely download HTTPS content from a URL, returning headers and body.

## Functions

# ca_trust_option()

    @spec ca_trust_option() ::
      {:cacerts, [:public_key.combined_cert()]}
      | {:cacertfile, String.t()}
      | no_return()

Resolve the trust source to hand to `:ssl` for peer verification.

Returns an SSL option pair — either `{:cacerts, list}` carrying DER-encoded certs from the platform trust store, or `{:cacertfile, path}` pointing at a PEM bundle on disk. The result is resolved once and cached in `:persistent_term`.

Precedence:

- an explicit `config :localize, cacertfile: path`,

- `:public_key.cacerts_get/0`,

- the `CAStore` hex package, if installed,

- the `:certifi` hex package, if installed,

- a well-known Unix file path that exists on disk.

Raises `Localize.NoCertificateStoreError` if none of these yield a usable trust source.

# certificate_locations()

    @spec certificate_locations() :: [String.t()]

Return all possible locations of a certificate file.

Returns dynamically discovered locations followed by well-known static locations on common operating systems.

### Returns

- A list of file path strings for all candidate certificate locations.

# dynamic_certificate_locations()

    @spec dynamic_certificate_locations() :: [String.t()]

Return the dynamically discovered certificate file locations.

These include application configuration and optional hex packages such as `CAStore` and `certifi`.

### Returns

- A list of file path strings for discovered certificate locations.

# get(url, options \\ [])

    @spec get(String.t() | {String.t(), list()}, options :: keyword()) ::
      {:ok, binary()} | {:not_modified, any()} | {:error, any()}

Securely download HTTPS content from a URL.

This function uses the built-in `:httpc` client but enables certificate verification which is not enabled by `:httpc` by default.

See also https://erlef.github.io/security-wg/secure_coding_and_deployment_hardening/ssl

### Arguments

- `url` is a binary URL or a `{url, list_of_headers}` tuple. If provided the headers are a list of `{'header_name', 'header_value'}` tuples. Note that the name and value are both charlists, not strings.

- `options` is a keyword list of options.

### Options

- `:verify_peer` is a boolean value indicating if peer verification should be done for this request. The default is `true` in which case the default `:ssl` options follow the erlef guidelines.

- `:timeout` is the number of milliseconds available for the request to complete. The default is "120000". This option may also be set with the `LOCALIZE_HTTP_TIMEOUT` environment variable.

- `:connection_timeout` is the number of milliseconds available for a connection to be established to the remote host. The default is "60000". This option may also be set with the `LOCALIZE_HTTP_CONNECTION_TIMEOUT` environment variable.

### Returns

- `{:ok, body}` if the return is successful.

- `{:not_modified, headers}` if the request would result in returning the same results as one matching an etag.

- `{:error, error}` if the download is unsuccessful. An error will also be logged in these cases.

### Unsafe HTTPS

If the environment variable `LOCALIZE_UNSAFE_HTTPS` is set to anything other than `"FALSE"`, `"false"`, `"nil"` or `"NIL"` then no peer verification of certificates is performed. Setting this variable is not recommended but may be required where peer verification fails for unidentified reasons.

### Certificate stores

In order to keep dependencies to a minimum, `get/2` attempts to locate an already installed certificate store. It will try to locate a store in the following order which is intended to satisfy most host systems. The certificate store is expected to be a path name on the host system.

    # A certificate store configured by the developer
    Application.get_env(:localize, :cacertfile)

    # Populated if hex package `CAStore` is configured
    CAStore.file_path()

    # Populated if hex package `certifi` is configured
    :certifi.cacertfile()

    # Debian/Ubuntu/Gentoo etc.
    "/etc/ssl/certs/ca-certificates.crt"

    # Fedora/RHEL 6
    "/etc/pki/tls/certs/ca-bundle.crt"

    # OpenSUSE
    "/etc/ssl/ca-bundle.pem"

    # OpenELEC
    "/etc/pki/tls/cacert.pem"

    # CentOS/RHEL 7
    "/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem"

    # OpenSSL on MacOS
    "/usr/local/etc/openssl/cert.pem"

    # MacOS & Alpine Linux
    "/etc/ssl/cert.pem"

# get_with_headers(request, options \\ [])

    @spec get_with_headers(String.t() | {String.t(), list()}, options :: keyword()) ::
      {:ok, list(), binary()} | {:not_modified, any()} | {:error, any()}

Securely download HTTPS content from a URL, returning headers and body.

This function uses the built-in `:httpc` client but enables certificate verification which is not enabled by `:httpc` by default.

See also https://erlef.github.io/security-wg/secure_coding_and_deployment_hardening/ssl

### Arguments

- `url` is a binary URL or a `{url, list_of_headers}` tuple. If provided the headers are a list of `{'header_name', 'header_value'}` tuples. Note that the name and value are both charlists, not strings.

- `options` is a keyword list of options.

### Options

- `:verify_peer` is a boolean value indicating if peer verification should be done for this request. The default is `true` in which case the default `:ssl` options follow the erlef guidelines.

- `:timeout` is the number of milliseconds available for the request to complete. The default is "120000". This option may also be set with the `LOCALIZE_HTTP_TIMEOUT` environment variable.

- `:connection_timeout` is the number of milliseconds available for a connection to be established to the remote host. The default is "60000". This option may also be set with the `LOCALIZE_HTTP_CONNECTION_TIMEOUT` environment variable.

- `:https_proxy` is the URL of an HTTPS proxy to be used. The default is `nil`.

### Returns

- `{:ok, headers, body}` if the return is successful.

- `{:not_modified, headers}` if the request would result in returning the same results as one matching an etag.

- `{:error, error}` if the download is unsuccessful. An error will also be logged in these cases.

### HTTPS Proxy

`Localize.Utils.Http.get_with_headers/2` will look for a proxy URL in the following locations in the order presented:

- `options[:https_proxy]`

- Localize compile-time configuration under the key `:localize[:https_proxy]`.

- The environment variable `HTTPS_PROXY`.

- The environment variable `https_proxy`.