The five S3 operations this proxy needs, over ex_aws_s3.
Not a general S3 client and not on the way to becoming one. An object-storage deployment needs exactly this much:
presign_get/4— a URL ffmpeg can read a source from, and a URL a client can be redirected to on a cache hit. Presigned because neither of them will send anAuthorizationheader for us.head/3— does this object exist, how big is it, what is its ETag. The/infoand cache-hit paths.put_stream/5— write a variant back, streaming, without knowing its length in advance (a render's output length is not known until it ends).get_stream/4— read a variant back for proxy-mode serving.delete/3— remove one object. Nothing on the request path deletes anything; this exists forAudioProxy.Config's boot-time writability probe, which proves a variant bucket accepts writes by performing one and then taking it back. A probe that only wrote would leave a small object behind on every restart.
Why a facade and not ExAws.S3 at the call sites
Three reasons, none of them dogma. Callers get {:error, :not_found}
instead of {:error, {:http_error, 404, _}}, which is the vocabulary the
rest of the codebase already speaks. Credentials come from
AudioProxy.Config rather than ex_aws's own resolution — see below. And
the surface stays four functions wide, so "we only do these four things"
is enforced rather than merely intended.
Configuration is ours, not ex_aws's
ex_aws resolves credentials from application env, AWS_* variables, and
on EC2 from IMDS, in an order it decides. This proxy validates its whole
configuration once at boot and keeps it in one map, so every call here
passes an explicit config override built from AudioProxy.Config. The
effect worth naming: there is no IMDS lookup, so an EC2 instance role
does not work and credentials must be supplied. That is a documented
limitation (README), not an oversight, and the place to change it is
config/1 here.
Addressing is configured, not inferred
A request names its bucket in the host (bucket.host/key, virtual-hosted)
or in the path (host/bucket/key, path-style). AP_S3_ADDRESSING picks,
defaulting to virtual-hosted with no custom endpoint and path-style with
one — AWS requires the former in regions launched after 2019, and every
S3-compatible store this project documents is deployed on the latter.
It is threaded into both places ex_aws reads it, which are not the same
place: the request path takes it from the config map, presigned_url/5
from its own options. See sign_get/4.
Two profiles, and :source is the default
Fetching a source and running the variant store are two jobs that a
deployment may hand to different providers or different principals, so
every function here takes the profile to run under as its last
argument: :source (the default, the AWS_*/AP_S3_* group) or :store
(that group with AP_VARIANT_S3_* overlaid). AudioProxy.Config builds
both and documents the fallback; all this module does is pick one.
Defaulting to :source rather than requiring the argument is deliberate:
the store is the small side — one module and the boot probe — and every
other caller is a source fetch, so an omitted profile is right far more
often than it is wrong.
Errors are data
Every function returns {:ok, _} or {:error, t:error/0}. :not_found
and :access_denied stay distinct: AWS answers 403 for a missing object
when the caller cannot list the bucket, so they are genuinely ambiguous —
but folding them would turn an expired credential into a permanent cache
miss and re-render every request forever.
Summary
Types
A bucket name.
Why an S3 call failed.
An object key, as raw bytes — never percent-encoded.
Which configuration profile an operation runs under.
Functions
The ex_aws config overrides for one profile of this deployment.
Reports whether credentials are configured.
Removes an object.
Streams an object's bytes, or an inclusive byte range of them.
Object existence, size, ETag and metadata.
A presigned GET URL for an object.
Uploads chunks — an enumerable of binaries of unknown total length.
Types
@type bucket() :: String.t()
A bucket name.
@type error() :: :not_found | :access_denied | :not_configured | :invalid_range | {:http, non_neg_integer(), binary()} | {:transport, term()}
Why an S3 call failed.
:not_found— the object is not there (404).:access_denied— credentials rejected (403). Distinct from:not_foundon purpose; see the moduledoc.{:http, status, body}— any other status.{:transport, reason}— nothing came back.:not_configured— no credentials. Refused here rather than passed toex_awsas nils, which it reads as "not provided" and answers by walking its own provider chain — ending at an instance-role lookup against 169.254.169.254 that, on a host where that address is not routed, hangs rather than failing.:invalid_range—get_stream/3only.
@type key() :: String.t()
An object key, as raw bytes — never percent-encoded.
@type object() :: %{ size: non_neg_integer(), etag: String.t() | nil, content_type: String.t() | nil, cache_control: String.t() | nil, metadata: %{optional(String.t()) => String.t()} }
What head/2 reports.
metadata holds the x-amz-meta-* headers with the prefix stripped and
names lowercased, which is how the variant store round-trips its own
response headers through an object.
@type profile() :: :source | :store
Which configuration profile an operation runs under.
:source is the shared AWS_*/AP_S3_* group; :store is that group with
the AP_VARIANT_S3_* overrides applied. See the moduledoc.
Functions
The ex_aws config overrides for one profile of this deployment.
Public so a test can assert on the addressing decision without issuing a request.
Reports whether credentials are configured.
Checked before every operation, because ex_aws treats a nil credential as
"not provided" rather than as an error — see error/0.
Removes an object.
A delete of a key that is not there is :ok — S3 answers 204 either way,
and the caller wanted the object gone rather than an inventory of what was
there first.
@spec get_stream( bucket(), key(), {non_neg_integer(), non_neg_integer()} | nil, profile() ) :: {:ok, Enumerable.t()} | {:error, error()}
Streams an object's bytes, or an inclusive byte range of them.
Lazy and bounded-memory: a sequence of ranged GETs of 1048576
bytes, never the whole object at once. ex_aws_s3 offers no in-memory
streaming read, so this is assembled here — see @read_chunk.
Only proxy-mode serving needs it. A redirect hands the client a presigned URL and the bytes never enter the BEAM at all, which is the default and the reason this costs more round trips than it might.
Object existence, size, ETag and metadata.
A presigned GET URL for an object.
Options: :expires_in (seconds, defaulting to AP_PRESIGN_TTL).
@spec put_stream(bucket(), key(), Enumerable.t(), keyword(), profile()) :: :ok | {:error, error() | term()}
Uploads chunks — an enumerable of binaries of unknown total length.
Options: :content_type, :cache_control, and :metadata (a map written
as x-amz-meta-*).
A stream that ends inside one part is a single PutObject; anything longer
is a multipart upload, aborted on every failure path so a failed write
leaves neither a partial object nor billable orphan parts.
Why the multipart protocol is driven here rather than by ExAws.S3.upload/4
Because upload/4 does not abort. ExAws.S3.Upload.perform/2 initiates,
uploads its parts, and on a part error simply returns that error — the
upload id goes out of scope and the initiated upload, with every part
already sent, stays in the bucket. ex_aws_s3 defines
abort_multipart_upload/3 and never calls it. Incomplete multipart uploads
do not appear in a bucket listing and are billed until a lifecycle rule
removes them, which makes this the one place in the S3 surface where
trusting the library costs money quietly.
upload/4 has a second problem on the same path: it collects task results
with Enum.map(fn {:ok, val} -> val end), which has no clause for the
{:exit, reason} a part-level timeout produces, so a slow part raises
FunctionClauseError from inside the dependency instead of returning an
error.
So the four operations are sequenced here. Everything underneath —
signing, request building, XML parsing, retries — is still ex_aws; what
is ours is the guarantee that an upload which does not complete is aborted. Parts go up one at a time, which keeps the memory
bound honest at one part plus one chunk and costs nothing worth having:
the render produces bytes far slower than S3 accepts them.
Operators should still set a lifecycle rule expiring incomplete multipart uploads. This code aborts on every path it can see; a hard kill of the VM is not one of them.