Performance

Kepler asks to be left on in production permanently. This is the argument for why that is reasonable, and the numbers behind it.

The trick

Telemetry handlers run inline in the process that emitted the event. A slow handler slows your checkout path directly, and a handler that allocates makes your request path allocate. So Kepler's handler does one thing: an atomic increment of a lock-free :counters slot. No allocation, no message send, no ETS write, no lookup.

A single poller then wakes on a tick, reads every counter, computes deltas, and evaluates every declared condition in one pass.

Event volume and evaluation volume are decoupled. 100k events per second costs 100k atomic increments plus one evaluation pass per second. Adding a watch adds work to the pass, not to your request path.

Measured cost

Measured on an Apple M4 Pro, 14 schedulers, Elixir 1.20 / OTP 29. Your numbers will differ; the shape will not. Reproduce with the script in this guide.

Per event, on your process

A filter that rejects is cheaper than any of these — it skips the increment and the captures entirely, which is the point of putting it first.

The number that matters is the third column. At 100k events per second, a counting watch costs about 5.5 ms of CPU per second — roughly 0.04% of one core on this machine, and 0.003% of a 14-core node.

Note what recent and enrich cost on a telemetry source. Each is an ETS write per event, and either roughly triples the handler. That is why both are opt-in, and why the guide says to reach for them deliberately rather than by default. On a report source, enrich is free — the report already contains the fields, and the process is dead, so there is nothing to introspect anyway.

A watch with no filter and no captures compiles to a two-element tuple the handler matches before incrementing, so the common case pays nothing for features it did not ask for.

Note also that :telemetry.execute/3 costs 40 ns before Kepler does anything. If you are already emitting the event, Kepler is a fraction of a cost you have already paid. If you are not, emitting it is the larger of the two decisions.

Per tick, on Kepler's own process

50 tier 2 watches — the most expensive kind, since they sample every tick — evaluate in 88 µs, about 1.8 µs per watch. At the default 1-second tick on a 14-scheduler node that is 0.0006% of the node's compute.

Telemetry watches are cheaper per tick than that unless they use percentiles, which read a 960-slot counter array and difference it. That is still microseconds.

Memory

A percentile watch allocates 960 counter slots, and :write_concurrency replicates them per scheduler — so the figure scales with your core count (960 × 14 × 8 bytes here). This buys lock-free writes under concurrency, which is the right trade for a hot path. If you declare dozens of percentile watches, this is the line item to look at.

The tier model

Every source belongs to a cost tier, and the declaration makes the tier obvious. Kepler.Watch records it; Kepler.watches/0 will tell you what you actually declared.

Tier 0 — free

Discrete sources that cost nothing until something happens.

source system_monitor: {:long_gc, 500} hands the VM a threshold and it enforces it. Zero steady-state cost; a message arrives only when it trips. There is no cheaper signal on the BEAM. The catch is that a node has exactly one system monitor, so Kepler detects an existing one and yields rather than stomping :observer or :recon.

source crash_report: :any and source supervisor_report: add a :logger filter that pattern-matches the report label. If nothing in your system is dying, that is a failed match per log event. Enrichment on these sources is free — the report OTP assembled already contains the fields, and the process is dead, so there is nothing to introspect even if you wanted to.

Tier 1 — near-free

source telemetry: [...]. One atomic increment per event, on your process. The handler is a captured named function with its counter reference passed in as handler config, so there is no runtime lookup — :telemetry's own ETS lookup happens once per event regardless of how many watches you declared on it.

Tier 2 — cheap, and constant

source process: and source vm:. A few reads per tick, independent of your traffic. Watches on the same process are read in one Process.info/2 call; watches on the same VM group share one read.

Kepler never calls :erlang.processes(). On a node with half a million processes that is ruinous at 1 Hz, and it is the single easiest way to turn a lightweight observer into the thing you are debugging. Watch named things precisely.

There is no tier 3

Tracing is the highest-risk surface and the least necessary, so it is out of scope. If you need it, :recon_trace already has the rate limiter and hard message cap that make it safe — use it directly, in a bounded window, and turn it off.

Self-budgeting

A library that asks to be always-on owes you a number rather than an assurance. Kepler measures its own cost every tick and reports it:

iex> Kepler.status().budget
%{
  share: 0.0006,               # fraction of node compute the pass consumed
  limit: 0.01,                 # what you configured
  level: 0,                    # 0 = everything running
  reductions_per_second: 4120, # Kepler's own processes
  schedulers: 14
}

share is tick duration / (window × schedulers) — measured with a monotonic clock around the evaluation pass, not modelled. reductions_per_second covers Kepler's processes as a whole, read per-process so it disturbs nothing.

Neither includes the telemetry handlers, because those run on your processes and are one atomic increment each. That is the point of the design: the cost that scales with your traffic is the cost that is too small to measure.

Every event payload carries the same numbers under "kepler", so the overhead travels with the notification.

Shedding

Going over budget for :shed_after consecutive ticks raises the shed level; staying under for :restore_after lowers it again. The hysteresis matters — shedding on a single slow tick would make Kepler flap alongside whatever made the node slow.

Tiers 0 and 1 are never shed. A system monitor costs nothing until it trips and a telemetry counter costs nothing to leave incrementing — turning either off would save nothing and lose data.

config :kepler,
  budget: [
    share: 0.01,          # 1% of the node; :infinity disables shedding
    shed_after: 3,
    restore_after: 10,
    max_tick: :timer.seconds(30)
  ]

Tuning

Raise the tick before you remove watches. Evaluation cost is per tick, so tick: 5_000 cuts it fivefold. The cost is resolution: sustained: windows and rate calculations become coarser, and a spike shorter than a tick is invisible.

Prefer :count and :average over percentiles when a mean would answer the question. A percentile watch costs 20 ns more per event and 115 KB.

Set keys: on every recent, and keep enrich lists short on telemetry sources. Telemetry metadata can be large, and both the ETS write and the webhook payload carry it.

Watch named processes, not many processes. A process: watch is cheap; a hundred of them on a hundred targets is a hundred Process.info/2 calls per tick.

Reproducing these numbers

# bench.exs — run with `mix run bench.exs`
defmodule Bench do
  use Kepler

  watch :counting do
    source telemetry: [:bench, :call]
    measure :count
    fire when: value > 1_000_000_000
  end
end

Application.stop(:kepler)
{:ok, _sup} = Kepler.Supervisor.start_link(watches: Bench, tick: 3_600_000, sinks: [])

measure = fn event ->
  {us, _} =
    :timer.tc(fn ->
      Enum.each(1..500_000, fn _i ->
        :telemetry.execute(event, %{duration: 1_234_567}, %{status: :ok})
      end)
    end)

  Float.round(us * 1000 / 500_000, 1)
end

IO.puts("no handler: #{measure.([:bench, :none])} ns")
IO.puts("watched:    #{measure.([:bench, :call])} ns")

For the evaluation pass, declare N watches, call Kepler.tick/0 in a loop, and read Kepler.status().budget.share.

What this does not cover

• Delivery. Webhook delivery is I/O in a supervised task off the poller. It does not affect the tick, and a wedged endpoint produces counted drops rather than back pressure. See sinks and payloads.
• Your telemetry events. If emitting the event is expensive, Kepler cannot make it cheaper.
• Clusters. Each node runs its own Kepler and does no cross-node correlation. That is a deliberate limit, not an oversight: distributed coordination in the hot path is how lightweight things stop being lightweight.