Configuring image_pipe_server

The server reads its configuration once at boot, from a TOML file and from environment variables. The guides describe what the settings do, with a tab for the server wherever its syntax differs.

An application that builds URLs for the server must match some of them, as listed in shared URL settings.

Sources and precedence

1. The library's defaults.
2. Standard AWS_ variables, for the S3 credential options that container_credentials and web_identity leave out (see S3 credentials).
3. The TOML file named by IPS_CONFIG, or /etc/image_pipe/config.toml. The default file is optional, so a deployment can use only environment variables. A missing file named by IPS_CONFIG stops the server.
4. IPS_ environment variables, which override single settings in the file.

A small file:

[url]
keys = ["0123abcd…"]

[sources.static]
adapter = "file"
match = "path"
root = "/data/images"
root_id = "static"

[sources.tmdb]
adapter = "http"
match = { prefix = "tmdb" }
base_url = "https://image.tmdb.org/t/p/original"
path_pattern = '[a-zA-Z0-9_-]+\.(jpg|jpeg|png|webp)'
stable = "immutable"
cache_policy = { storage = "allow" }

[cache.output]
root = "/var/cache/image_pipe/output"
max_size_bytes = 10_000_000_000
node_id = "node-0"

[processing]
quality = 82

[pool]
max_concurrency = 8
max_queue = 16

Environment variables

A variable names a setting by its path in the file: IPS_, then the levels joined with __ and written in upper case. Single underscores stay part of a name.

IPS_URL__KEYS=0123abcd…
IPS_SOURCES__TMDB__ADAPTER=http
IPS_SOURCES__TMDB__MATCH__PREFIX=tmdb
IPS_SOURCES__TMDB__BASE_URL=https://image.tmdb.org/t/p/original
IPS_PROCESSING__QUALITY=82

• Levels are lowercased, so a source written as [sources.TMDB] in the file can't be overridden from the environment. Use lowercase source names.
• Lists are comma-separated, such as IPS_URL__KEYS=0123abcd…,4567ef01…, or have one entry per line.
• A variable ending in _FILE reads the value from that file, without trailing whitespace, for Docker and Kubernetes secrets: IPS_URL__KEYS_FILE=/run/secrets/signing_keys. A secret file can hold one list entry per line.
• Setting the same value twice is an error, such as IPS_URL__KEYS together with IPS_URL__KEYS_FILE, or with IPS_url__keys, since levels are lowercased.
• A list variable with no entries, or a _FILE variable whose file has none, is an error, so an empty signing-keys secret stops the server instead of turning off signature checks.
• Settings whose own name ends in _file, such as the web_identity provider's token_file, take the variable's value as the path instead: IPS_SOURCES__MEDIA__CREDENTIALS__TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token.
• Variables that Kubernetes sets for a Service named ips or ips-…, such as IPS_SERVICE_HOST and IPS_PORT, are ignored. Every setting has a __ in its name, and these never do.
• Settings that are awkward as variables, such as S3 buckets, address_policy, or storage_inputs, belong in the file.

Values

TOML strings, numbers, booleans, arrays, and tables map onto the library's options:

• Named values are strings: stable = "immutable", http_cache = "auto".
• Tagged values are tables with one entry: freshness = { fallback = 60 }.
• Tables with library-defined keys, such as format_quality = { webp = 80 }, accept only keys the library knows.
• adapter names a built-in source adapter: "file", "http", or "s3".
• match is "path" or a table of prefix and scheme rules, each a string or an array.
• path_pattern is a regular expression, anchored by the adapter.
• HTTP sources take request_headers (a table of header names to values) and bearer_token for origins behind an API key or a static token. Both can come from _FILE variables. A shell can't export a variable named after a header with a -, such as IPS_SOURCES__TMDB__REQUEST_HEADERS__X-API-KEY_FILE. Docker, Compose, and Kubernetes accept the name, so set it there. Otherwise, set the header under request_headers in the TOML file.
• S3 credentials are { static = { access_key_id = "…", secret_access_key = "…" } } or a provider: { provider = "instance_role" }, "container_credentials", "web_identity", or "assume_role", with the provider's options in the same table.

Some settings exist only in Elixir: functions (address_resolver, the function form of address_policy, clock), req_options, custom source adapters, detectors, and credential providers. The reference marks them. Hosts that need them build their own release on top of image_pipe.

Errors

Invalid configuration stops the server at boot. The server prints the error, which names the setting or variable, and exits with status 1. Errors about the file's shape and types never quote a value. The library's own checks may quote a non-secret value, such as an out-of-range quality, but never a key, credential, token, or the contents of a _FILE:

invalid configuration: url.source_encryption_keys[0]: expected a hex-encoded 32-byte key
invalid configuration: processing.qualty: unknown setting

A file that isn't valid TOML is reported with its position and key, with the values on the quoted line redacted:

invalid TOML: unexpected newline in single-quoted string in /etc/image_pipe/config.toml on line 2, column 14:

    keys = <redacted value>
           ^

Reference

Every key the server accepts, generated from the loader's schemas. Types are TOML types; defaults are the library's.

[server]

The HTTP listener. Times are in milliseconds. read_timeout closes connections that send nothing for that long, idle keep-alive connections included. max_connections rounds up to a multiple of 100 above 100 connections. With auth_token, requests other than the health checks must send Authorization: Bearer <token>. On shutdown, /health/ready answers 503 while the server keeps serving for shutdown_delay. Then in-flight requests get shutdown_timeout to finish. health_port serves the health checks on a separate listener, outside max_connections. See health and shutdown.

[url]

How the server checks request URLs: the signing keys and the keys that decrypt enc/ image paths. See signing URLs and rotating keys.

[sources.<name>]

One table per source. The table name is the source's name. Besides adapter and match, a source takes its adapter's options.

adapter = "file"

adapter = "http"

Elixir only: req_options, address_resolver.

adapter = "s3"

Elixir only: req_options.

S3 credentials

Static credentials:

Or a credential provider and its options. assume_role takes base credentials in the same forms. container_credentials without relative_uri, full_uri, auth_token, or auth_token_file takes each one whose variable is set from AWS_CONTAINER_CREDENTIALS_RELATIVE_URI, AWS_CONTAINER_CREDENTIALS_FULL_URI, AWS_CONTAINER_AUTHORIZATION_TOKEN, and AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE. web_identity fills each of token_file, role_arn, region, and role_session_name that the table leaves out, from AWS_WEB_IDENTITY_TOKEN_FILE, AWS_ROLE_ARN, AWS_REGION, and AWS_ROLE_SESSION_NAME respectively.

provider = "assume_role":

provider = "container_credentials":

provider = "instance_role":

provider = "web_identity":

[cache]

Caches on the local filesystem: output for processed images, input for originals. Their root directories must be separate, neither inside the other. Setting max_size_bytes bounds a cache and requires node_id. See caching processed images.

[processing]

Defaults and limits for every image the server processes. The limits on originals, intermediate images, and results are described in limiting work per request.

Elixir only: max_preset_lookups, preset_lookup, telemetry_prefix, clock.

[pool]

How many images are processed at once, and how long requests wait for a turn. Times are in milliseconds. max_concurrency defaults to the number of CPU cores the server can use. See limiting concurrent processing.

[http]

Response settings: the CORS origin, whether requests may ask for debug headers, and the HTTP cache headers. See serving images through a CDN.

[telemetry]

log_level is the lowest level the server logs. log_requests logs each request and its stages at info, and failures and degraded results at warning. With trust_traceparent, a request with a W3C traceparent header joins the caller's trace when tracing is on. Any client can send that header. With trust_request_id, an incoming x-request-id of 1 to 200 letters, digits, and -_.:+/= tags the request's log lines instead of a generated ID.