# Rendering

How a render runs: the subprocess, the chunk stream it produces, and the rules
that bound it. The argument vector handed to that subprocess is a separate
subject — see [ffmpeg-arguments.md](ffmpeg-arguments.md).

`AudioProxy.Ffmpeg.Render` is one GenServer per render, under
`AudioProxy.Ffmpeg.RenderSupervisor`. It owns exactly one ffmpeg process and
exists for exactly as long as that process does.

## The consumer contract

A render is started with an argv list and a consumer process. The consumer
receives three kinds of message, and `render` is the render's pid:

| Message | Meaning |
|---|---|
| `{:chunk, render, binary}` | stdout bytes, in order |
| `{:done, render, %{exit_status: 0}}` | clean exit, every byte delivered |
| `{:error, render, reason}` | anything else |

Exactly one `{:done, …}` or `{:error, …}` is sent, always after the last chunk,
and the render stops immediately afterwards. A consumer that wants to hear
about a render that *crashed* — as opposed to one that failed — monitors it,
because a crash sends no message at all.

This contract is the seam the rest of the system is built on. Coalescing
subscribes several requests to one render's chunk stream; chunked HTTP delivery
and the S3 write-back both consume it; `/info` reuses the same subprocess
plumbing for `ffprobe`. None of them know what ffmpeg is.

## Why a subprocess at all

ffmpeg does every byte of decoding and encoding; the Elixir side only
orchestrates. That is a licensing posture as much as an architectural one — the
(L)GPL boundary is a *process* boundary, so invoking a CLI keeps even a
GPL-configured ffmpeg from reaching this source tree.

The subprocess is spawned from an argv list with
`Port.open({:spawn_executable, …}, args: argv)`. There is no shell anywhere in
the path, which is what makes a source URL containing `;`, `$(…)`, a quote or a
space simply one argument. The injection-safety property that
`AudioProxy.Ffmpeg.Command` is written for only holds because nothing between
it and `execve` re-parses its output.

## Buffering, and what it is not

Ports have no passive read mode. The VM drains the subprocess' stdout pipe as
fast as the OS fills it and mails the bytes to the render process, whether or
not anything downstream is ready for them. Left alone, a slow client would
therefore turn into an unbounded mailbox.

So the render accounts *outstanding* bytes — forwarded to the consumer but not
yet acknowledged with `ack/2`. Above a high-water mark of 1 MiB it stops
forwarding and queues internally; ffmpeg fills the ~64 KB OS pipe and blocks on
its own write. Acknowledging drops the count and releases the queue. A consumer
that never acknowledges receives the high-water mark plus at most one chunk, and
then nothing further — including the completion message, which is deliberately
withheld until every chunk before it has been delivered.

**This is a bounded buffer, not true backpressure.** Between the high-water mark
and ffmpeg actually blocking there is a pipe's worth of slack, and whatever the
render has queued is still held in memory. For preview-sized outputs — the
sizes this proxy is built for — that is the right trade: no extra moving parts,
no dependencies, and a bound that a container can multiply by
`AP_MAX_CONCURRENCY` and reason about.

The escalation, if full-length transcodes ever need it, is the named-pipe
pattern: ffmpeg writes to a `mkfifo`, and Elixir reads it passively with
`IO.binread` in raw mode, where the OS pipe blocking *is* the backpressure (file
reads have run on dirty I/O schedulers since OTP 21). It is a change of
mechanism behind the same consumer contract, which is why it can wait until
there is a measurement asking for it.

**In the running system this high-water mark is not what bounds memory**, and
the reason is the next section. The render's only consumer is the coalescing
coordinator, which retains every byte anyway and therefore acknowledges each
chunk the moment it arrives. The bound that applies is the coordinator's
retention cap. The mechanism above is still what protects a *direct* consumer
of the contract, and is what the FIFO escalation replaces.

## Render policy

A render runs at full speed; the client's chunked stream lags it rather than
throttling it. A client listening in real time must not be able to pin a CPU
slot for the duration of the audio.

The write-back this policy was written for does not exist yet — there is no
variant bucket and no tee, so today "full speed" means only that the encoder is
never paced by the socket. The policy is what the write-back will land into
(`add-variant-cache`), not something already running.

## Lifecycle: no ffmpeg outlives its render

This is the guarantee the module is arranged around, because the failure it
prevents is the expensive one: an orphaned encoder holds a CPU slot, and a
proxy that leaks one per cancelled request degrades until it is restarted.

Every way a render can end — a clean finish, `cancel/1`, the timeout, a dead
consumer, the supervisor shutting down at VM stop — arrives at the same
`terminate/2`, which is why exits are trapped: shutdown becomes one of the
ordinary paths rather than an exception to them. From there:

1. close the port,
2. `SIGTERM` the subprocess,
3. `SIGKILL` after a two-second grace, if it is still there.

**Closing the port is not enough on its own**, and that is the whole reason for
steps 2 and 3. The BEAM does not signal the process on the far side of a closed
port. An ffmpeg blocked reading a slow HTTP input may not touch its stdout for
minutes, never notice that nobody is listening, and sit there holding a slot.
Measured on this project: a `SIGTERM`-ignoring subprocess survives
`Port.close/1` indefinitely.

The grace is a trade. Two seconds is long enough for ffmpeg to flush and close
cleanly, short enough that a cancelled request is not noticeably held open.
Within that window a PID could in principle be reused and the `SIGKILL` land on
an innocent process; on any real system the window makes that implausible, and
the alternative — never escalating — is the orphan this exists to prevent.

## Timeout

A render exceeding `AP_RENDER_TIMEOUT` (seconds, default 300) is killed by the
same discipline and reported as `:timeout`, which the HTTP layer maps to 504.
The timer is armed at spawn from configuration, so an operator raising the
limit for long masters changes one environment variable and nothing else.

## Failure classification

ffmpeg exits 1 for almost everything, so the exit status alone cannot separate
"the file isn't there" from "the file isn't audio" — and those want different
HTTP statuses. The class therefore comes from matching a bounded tail of
stderr:

| Class | Recognised from | HTTP |
|---|---|---|
| `:not_found` | `No such file or directory`, `Server returned 404`, `403`/`Forbidden` | 404 |
| `:undecodable` | `Invalid data found when processing input`, `could not find codec parameters` | 415 |
| `:timeout` | the render timer, not stderr | 504 |
| `:cancelled` | `cancel/1` | — the client has already gone |
| `:render_failed` | anything else | 500 |

The consumer receives `%{class: _, exit_status: _, stderr: _}`. Keeping the raw
tail alongside the class matters: the class is what the proxy acts on, the tail
is what an operator needs when the class is `:render_failed` and the question
is *why*.

stderr goes to a per-render file in a scratch directory, never merged into
stdout — merging would splice diagnostics into the audio. Only the last 4 KiB
is read back, so a decoder in a complaining mood cannot turn a failed render
into a memory problem. Each render deletes its own file, and the supervisor
sweeps the directory at boot for the renders that died with the VM.

One note on how that file is arranged, since it is the only place a shell
appears in a project that otherwise forbids them. Erlang ports offer stdout, or
stdout with stderr merged in, and nothing else: there is no port option for
redirecting stderr to a file. So the subprocess is spawned as

```
/bin/sh -c 'exec "$0" "$@" 2>"$AP_RENDER_STDERR"' <binary> <args…>
```

The script is a compile-time constant with no user data in it. The binary
arrives as `$0`, its arguments as `"$@"`, and the path through the environment,
so a source URL full of metacharacters is quoted shell *data* and never shell
text. `exec` then replaces the shell with ffmpeg, which means the pid the kill
discipline signals is ffmpeg's own and there is no intermediate process left to
orphan.

## Coalescing: one render per cache key

Requests do not reach `AudioProxy.Ffmpeg.Render` directly. They go through
`AudioProxy.RenderCoordinator`, which is a single-flight in front of it: a
`Registry` of unique cache keys, one coordinator process per key, and a
`DynamicSupervisor` holding them. Twenty simultaneous requests for the same
variant run one ffmpeg, not twenty.

Subscribing and starting are the same operation, which is what makes the start
race a non-event. `DynamicSupervisor.start_child/2` either succeeds — this
caller started the render, and is `MISS` — or answers `{:error, {:already_started,
pid}}`, which is `COALESCED` and a join. Nobody has to check whether a render
exists before deciding to start one.

The coordinator broadcasts the consumer contract above, with itself as the
handle, so a consumer written against the render works against the coordinator
unchanged. Two calls differ, because they name a handle:

| Render | Coordinator | Why |
|---|---|---|
| `ack/2` | — | the coordinator retains the bytes and acknowledges for you |
| `cancel/1` | `unsubscribe/1` | one subscriber leaving must not cancel a render the others are reading |

**Late joiners are handed a backlog.** Everything produced so far, in order,
returned by the same call that registers the subscriber — one coordinator
callback, so there is no window in which a chunk could be both broadcast and
backlogged, or neither. The consumer writes the backlog, then the live chunks.

**Retention is the memory bound.** The bytes are held anyway (the variant-cache
slice tees the same ones to storage), so what stops a render growing without
limit is a cap: output past `AP_MAX_VARIANT_BYTES` fails the render for every
subscriber. That ceiling is the variant's own, separate from `AP_MAX_SRC_BYTES`
and defaulting to it, so a deployment can accept long masters and still bound
one render's retention to a preview's worth of bytes. The breach is only
knowable once the response has committed to `200`, so it is a failed stream
rather than a `413` — which is the reason the FIFO escalation above is written
down; spooling instead of retaining changes nothing in the contract.

Three ways a coordinator ends, differing in what happens to the key:

- **Done** — the completion is broadcast, then it lingers about a second, still
  registered, so a request that arrives just too late gets the finished bytes
  instead of re-rendering. A straggler past the linger starts a fresh render:
  wasteful, never wrong.
- **Failure** — broadcast to every current subscriber, then the key is
  unregistered *immediately*, so the next request retries rather than attaching
  to a corpse.
- **Last subscriber gone** — nobody is listening, so the subprocess is
  cancelled. `unsubscribe/1` returns only once that has happened, which makes
  it a barrier the way `cancel/1` is. One subscriber dying among several is
  just a removal; the render continues.

## Slots: `AP_MAX_CONCURRENCY` and the wait queue

Coalescing bounds the worst hot-key case. What it cannot bound is a burst across
*distinct* keys, and that is what `AudioProxy.Semaphore` is for: a counting
semaphore of `AP_MAX_CONCURRENCY` slots with a FIFO wait queue of
`AP_QUEUE_SIZE` behind it.

**A slot is per render, not per request.** The coordinator takes one before it
spawns its pipeline and releases it from `terminate/2`, so twenty requests
coalescing on one key cost one slot, and the slot's lifetime is the subprocess'
lifetime — including the kill discipline, since the release happens *after* the
cancel rather than before it. A slot handed over while the previous ffmpeg was
still being SIGKILLed would exceed the cap by exactly the margin that takes.

**The coordinator waits without blocking.** It asks with `Semaphore.request/1`,
which answers immediately with `:granted`, `:queued` or a queue-full error, and
starts its render when the grant arrives as a message. So a coordinator has one
phase more than the render does — `:queued`, before `:rendering` — and keeps
answering joins throughout it. That matters: the requests coalescing onto a
queued render are exactly the ones that must not each take a slot of their own.
A subscriber cannot tell the two phases apart, and has no reason to. It is
waiting for bytes either way.

**A full queue is a start failure, not a render failure.** The slot is asked for
in the coordinator's `init/1`, so `subscribe/2` answers
`{:error, {:queue_full, retry_after}}` and no coordinator is left registered
under that key — the next request asks the semaphore again instead of joining
something that is never going to render. `AudioProxy.ErrorJSON` renders that
tuple as a `429` with `Retry-After`, which is the only place that header comes
from.

**A wait that runs out is the same 429, not a 504.** The queue being *full* and
the queue being *too slow* are the same answer from the client's side, so they
get the same one. The render endpoint waits for `{:rendering, _}` — the
coordinator's announcement that it has a slot and has started — under the same
budget it later gives the render itself, and expiring before that message is
429 with a fresh `Retry-After` from the semaphore. Expiring after it is 504,
which now means what it says: a render exists and has gone silent. Before the
split, a request that merely queued too long was told "Render exceeded
AP_RENDER_TIMEOUT" about a render that had never started, and the request log
repeated it.

That budget is also what bounds the wait. Letting a queued request wait
indefinitely would hold the connection — and the queue place behind it — for as
long as the queue took.

The estimate on a rejection is a moving average of recent slot-hold durations,
scaled by how deep the queue already is: roughly "how long the renders in front
of you have been taking, times how many of them there are, over how many run at
once",
clamped to at least a second and at most `AP_RENDER_TIMEOUT`. Coarse by
construction — a client that comes back early finds the queue full and is told
again.

**Slots are recovered by monitor, not by discipline.** Every holder and every
waiter is monitored: a coordinator that crashes releases its slot when the
`DOWN` lands, and a waiter that dies is dropped from the queue rather than
granted one nobody is holding. `release/1` is a promptness optimisation on top
of that, and is idempotent. The path this exists for is the render that *hangs*
rather than fails — a stalled origin, an input that never yields a byte — where
nothing but `AP_RENDER_TIMEOUT` ends it: the timeout fails the render, the
coordinator stops, and the slot goes to the next waiter.

Occupancy and queue depth are published as `[:audio_proxy, :semaphore, _]`
telemetry events (`acquired`, `queued`, `rejected`, `released`, `abandoned`).
`AudioProxy.Metrics` counts `rejected` from that set, and reads the gauges off
`Semaphore.stats/2` per scrape rather than from the events — a gauge
maintained by events needs its increments and decrements to balance for the
life of the VM, and asking the semaphore what it holds cannot drift. Either
way this path is untouched.

## Delivery over HTTP

The render endpoint (`AudioProxy.Plugs.RenderAction`) is a consumer of the
contract above and nothing more: it subscribes to the coordinator for its cache
key, writes the catch-up backlog if it was handed one, then loops on the mailbox
writing each chunk with `Plug.Conn.chunk/2`. It reports which it was in
`X-Audio-Proxy`: `MISS` for the request that started the render, `COALESCED` for
one that attached to it.

**Before and after the first byte are different worlds.** Before it, nothing has
been sent and a failure is an ordinary JSON error: the class maps to 404, 415,
500 or 504. After it, the status line is spent, and the only signal HTTP/1.1
leaves is an abnormal close — the connection is torn down without the
terminating chunk, which is what §5 means by "abnormal termination of the
chunked stream". A client that treats a truncated chunked response as a
complete file will believe a failed render succeeded.

**Disconnect is detected by writing.** `chunk/2` answering `{:error, _}` means
the socket is gone, and the request unsubscribes on the spot; whether the render
stops is the coordinator's call, since another client may still be reading the
same variant. That is not the only guarantee — the coordinator monitors its
subscribers and the render monitors its consumer — but it is the prompt one. It
costs a caveat: detection happens on the
*next* chunk, so teardown is bounded by chunk cadence rather than by wall clock.
For an encoder producing output continuously that is milliseconds; for one
stalled on a slow input it is the render timeout, which is the same bound that
applies to a client still listening.

**The endpoint's own deadline is a backstop.** `AP_RENDER_TIMEOUT` is enforced
by the render process, which owns the subprocess and can classify the failure.
The endpoint applies the same budget to its own mailbox, slightly wider, so that
a render dying without a word cannot leave a request hanging — but the timeout a
client normally sees is the render's, with its class intact.