The write path. Accumulate in memory, flush in batches, drop under pressure.
GoatCounter's memstore, in BEAM terms: hits land in a buffer, a bulk insert drains it on a size or time trigger, and nothing on the request path ever waits for a database. That design does 800 hits/sec on a $5 VPS, which is the bar this has to clear.
Flush policy
flush_bytes accumulated or flush_ms elapsed, whichever comes first
(100KB / 5s by default — Plausible's numbers). Size alone stalls a quiet site
forever; time alone makes a busy site do a round-trip per event.
Dropping is the correct failure
When the store is slow or down, the buffer grows. Three lines of defence, in
order: at most @max_inflight flushes run at once, so a slow store cannot
spawn unbounded work; past max_buffer events new arrivals are dropped;
and a failed flush returns its batch to the front of the buffer exactly once,
then gives up on it.
Losing analytics is a bad day. Growing a queue until the node dies takes the
application down with it, and the application is the thing that matters.
Every drop emits [:pixelex, :ingest, :drop] — silent loss is the actual
danger here, not loss.
Telemetry
[:pixelex, :ingest, :push]—%{count: 1}[:pixelex, :ingest, :flush]—%{count:, bytes:, duration_us:, written:}[:pixelex, :ingest, :drop]—%{count:},%{reason: :buffer_full | :flush_failed}
Ceiling
One GenServer serialises every push. At a list prepend per message that is
well past a million events/sec — three orders of magnitude above the target —
so it is not worth partitioning yet. ponytail: single buffer process; move
accumulation to :ets with write_concurrency and an :atomics byte counter
if a profile ever shows this mailbox as the bottleneck.
Summary
Functions
Returns a specification to start this module under a supervisor.
Flush now and wait for it. Tests and graceful shutdown only — calling this on a request path reintroduces exactly the coupling this module exists to avoid.
Buffered event count.
Queue one event. Returns :ok immediately, always.
Queue a batch.
%{pending: n, inflight: n} — buffered events, and flushes currently in
flight.
Functions
Returns a specification to start this module under a supervisor.
See Supervisor.
@spec flush(timeout()) :: {:ok, non_neg_integer()} | {:error, term()}
Flush now and wait for it. Tests and graceful shutdown only — calling this on a request path reintroduces exactly the coupling this module exists to avoid.
@spec pending() :: non_neg_integer()
Buffered event count.
@spec push(Pixelex.Event.t()) :: :ok
Queue one event. Returns :ok immediately, always.
A cast, deliberately: the caller is a request that must not wait on
analytics, and a call here would put the buffer's health on the critical
path of every page load.
@spec push_all([Pixelex.Event.t()]) :: :ok
Queue a batch.
@spec stats() :: %{pending: non_neg_integer(), inflight: non_neg_integer()}
%{pending: n, inflight: n} — buffered events, and flushes currently in
flight.
inflight is the number that matters when something is wrong: it sits at
5 when the store has stopped acknowledging writes, which is
the signal that arrives before the buffer fills and the drops start.