Pixelex.Ingest (Pixelex v0.1.0)

Copy Markdown View Source

The write path. Accumulate in memory, flush in batches, drop under pressure.

GoatCounter's memstore, in BEAM terms: hits land in a buffer, a bulk insert drains it on a size or time trigger, and nothing on the request path ever waits for a database. That design does 800 hits/sec on a $5 VPS, which is the bar this has to clear.

Flush policy

flush_bytes accumulated or flush_ms elapsed, whichever comes first (100KB / 5s by default — Plausible's numbers). Size alone stalls a quiet site forever; time alone makes a busy site do a round-trip per event.

Dropping is the correct failure

When the store is slow or down, the buffer grows. Three lines of defence, in order: at most @max_inflight flushes run at once, so a slow store cannot spawn unbounded work; past max_buffer events new arrivals are dropped; and a failed flush returns its batch to the front of the buffer exactly once, then gives up on it.

Losing analytics is a bad day. Growing a queue until the node dies takes the application down with it, and the application is the thing that matters. Every drop emits [:pixelex, :ingest, :drop] — silent loss is the actual danger here, not loss.

Telemetry

  • [:pixelex, :ingest, :push]%{count: 1}
  • [:pixelex, :ingest, :flush]%{count:, bytes:, duration_us:, written:}
  • [:pixelex, :ingest, :drop]%{count:}, %{reason: :buffer_full | :flush_failed}

Ceiling

One GenServer serialises every push. At a list prepend per message that is well past a million events/sec — three orders of magnitude above the target — so it is not worth partitioning yet. ponytail: single buffer process; move accumulation to :ets with write_concurrency and an :atomics byte counter if a profile ever shows this mailbox as the bottleneck.

Summary

Functions

Returns a specification to start this module under a supervisor.

Flush now and wait for it. Tests and graceful shutdown only — calling this on a request path reintroduces exactly the coupling this module exists to avoid.

Buffered event count.

Queue one event. Returns :ok immediately, always.

Queue a batch.

%{pending: n, inflight: n} — buffered events, and flushes currently in flight.

Functions

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

See Supervisor.

flush(timeout \\ 15000)

@spec flush(timeout()) :: {:ok, non_neg_integer()} | {:error, term()}

Flush now and wait for it. Tests and graceful shutdown only — calling this on a request path reintroduces exactly the coupling this module exists to avoid.

pending()

@spec pending() :: non_neg_integer()

Buffered event count.

push(event)

@spec push(Pixelex.Event.t()) :: :ok

Queue one event. Returns :ok immediately, always.

A cast, deliberately: the caller is a request that must not wait on analytics, and a call here would put the buffer's health on the critical path of every page load.

push_all(events)

@spec push_all([Pixelex.Event.t()]) :: :ok

Queue a batch.

start_link(opts)

stats()

@spec stats() :: %{pending: non_neg_integer(), inflight: non_neg_integer()}

%{pending: n, inflight: n} — buffered events, and flushes currently in flight.

inflight is the number that matters when something is wrong: it sits at 5 when the store has stopped acknowledging writes, which is the signal that arrives before the buffer fills and the drops start.