The four operator signals, aggregated from telemetry and exposed to a scraper: saturation, latency, cache efficiency, errors.
A GenServer owning one ETS table, plus a :telemetry handler that writes to
it. The process is not in the write path — a handler runs in whichever
process emitted the event and updates the table directly — so instrumenting
a render costs an :ets.update_counter/4 and no message. What the process is
for is ownership: the table lives and dies with it, and terminate/2 detaches
the handler so nothing is left writing into a table that has gone.
Why this and not prom_ex
The dependency policy, and the same argument add-s3-client reached the
opposite conclusion on. There the volume was decisive — ~2000 lines of
signing against a contract AWS controls. Here the metric set is a dozen
fixed series and the exposition format has been stable since 2014, so the
whole of it is AudioProxy.Metrics.Exposition and the table below. A
Prometheus library would bring a Phoenix-oriented dependency tree to save
code this project can read in one sitting.
Counted, and sampled
Two kinds of metric, and the split is not cosmetic.
Counters and the histogram are event-driven and accumulate in ETS. They
are monotonic, so a concurrent update is :ets.update_counter/4 and exact,
and nothing is lost by reading them late.
Gauges are sampled at scrape time, from AudioProxy.Semaphore.stats/2
and the coalescing registry, not from events. A gauge maintained by events
would need an increment and a decrement to balance for the life of the VM,
and one abnormal exit — a Bandit connection process killed outright with a
render attached — leaves it permanently wrong with nothing to correct it.
Asking the semaphore what it holds cannot drift, because the semaphore is
the answer. The cost is a GenServer.call per scrape, bounded by
@sample_timeout; a semaphore that cannot answer within it omits its four
gauges from that scrape rather than delaying it, which a scraper reads as a
gap and an operator reads as the semaphore being the problem.
What is exported
| Metric | Type | Labels | Source |
|---|---|---|---|
audio_proxy_renders_total | counter | format, outcome | render stop / exception |
audio_proxy_render_duration_seconds | histogram | format, outcome | render stop / exception |
audio_proxy_renders_running | gauge | — | coalescing registry |
audio_proxy_render_slots_held | gauge | — | semaphore |
audio_proxy_render_slots_capacity | gauge | — | semaphore |
audio_proxy_render_queue_depth | gauge | — | semaphore |
audio_proxy_render_queue_capacity | gauge | — | semaphore |
audio_proxy_render_queue_rejections_total | counter | — | semaphore :rejected |
audio_proxy_cache_lookups_total | counter | format, outcome | cache lookup |
audio_proxy_variant_store_write_failures_total | counter | — | write-back tee |
audio_proxy_http_requests_total | counter | endpoint, status | Bandit request stop |
outcome on a render is success for one the client received whole,
cancelled for one abandoned because the client went away, and otherwise
the failure class (timeout, undecodable, not_found, …). outcome on a
cache lookup is §5's three: hit, miss, coalesced. status is a code
family (2xx, 4xx, …) and unknown for a request that died before it had
one — Bandit emits its stop event either way, and AudioProxy.LogHandler
has the story of why that is not hypothetical.
Label discipline is a hard rule, not a style
Every label value here comes from a bounded enum this codebase defines — a format, an outcome, an endpoint class, a status family. Nothing derived from a request reaches a label: not the source, not the options string, not the cache key. A label whose values a client chooses is a series count a client chooses, and the failure mode is the scraper's storage rather than this process' — which is exactly why the rule belongs at the point of instrumentation and not in the scraper's config.
renders_running counts coordinators, so it counts renders and not
requests: twenty clients coalesced onto one encode are one running render,
which is the same thing AP_MAX_CONCURRENCY counts.
A handler that raises is a handler that stops existing
The same hazard AudioProxy.LogHandler documents, and the same defence:
:telemetry detaches a raising handler permanently and silently, so one
malformed event would cost every future metric rather than one. Every read
of an event's payload falls back rather than matching, and handle_event/4
rescues on top of that.
Summary
Functions
The bucket edges of audio_proxy_render_duration_seconds, in seconds.
Returns a specification to start this module under a supervisor.
Every metric this module exports, in exposition order.
Discards every counted value. For tests only.
Renders the current state as a Prometheus exposition.
Starts the aggregator, creating its table and attaching its handler.
Functions
@spec buckets() :: [float()]
The bucket edges of audio_proxy_render_duration_seconds, in seconds.
Returns a specification to start this module under a supervisor.
See Supervisor.
@spec definitions() :: [AudioProxy.Metrics.Exposition.definition()]
Every metric this module exports, in exposition order.
Public so that a test can assert the set has not changed by accident, and so that the README's table has one source it can be checked against.
@spec reset() :: :ok
Discards every counted value. For tests only.
Sampled gauges are unaffected — there is nothing here to reset, since they are read from the semaphore and the registry at scrape time.
Not atomic, and deliberately left that way. Clearing and re-seeding are two
operations, and an event landing between them creates a counter that
seed/0's insert_new will not then flatten — so that series survives the
reset at 1. Every caller is a test, the suite's metrics tests are
async: false, and a lock around a test helper would be machinery for a
race that production cannot reach: nothing on the request path calls this.
@spec scrape() :: iodata()
Renders the current state as a Prometheus exposition.
Samples the gauges as it goes, so what it returns is a snapshot taken now rather than a replay of the last event.
@spec start_link(keyword()) :: GenServer.on_start()
Starts the aggregator, creating its table and attaching its handler.