# Performance

Kepler asks to be left on in production permanently. This is the argument for
why that is reasonable, and the numbers behind it.

## The trick

Telemetry handlers run **inline in the process that emitted the event**. A slow
handler slows your checkout path directly, and a handler that allocates makes
your request path allocate. So Kepler's handler does one thing: an atomic
increment of a lock-free `:counters` slot. No allocation, no message send, no
ETS write, no lookup.

A single poller then wakes on a tick, reads every counter, computes deltas, and
evaluates every declared condition in one pass.

**Event volume and evaluation volume are decoupled.** 100k events per second
costs 100k atomic increments plus one evaluation pass per second. Adding a
watch adds work to the pass, not to your request path.

## Measured cost

Measured on an Apple M4 Pro, 14 schedulers, Elixir 1.20 / OTP 29. Your numbers
will differ; the shape will not. Reproduce with the script in this guide.

### Per event, on your process

| | ns/event | Added by Kepler |
| --- | ---: | ---: |
| `:telemetry.execute/3` with no handler attached | 40 | — |
| `measure :count` | 92 | **52** |
| `measure :duration, percentile: 99` | 112 | **72** |
| `measure :count` + a `filter` that passes | 108 | **68** |
| `measure :count` + `enrich [:status]` | 198 | **158** |
| `measure :count` + `recent size: 25` | 232 | **192** |

A `filter` that *rejects* is cheaper than any of these — it skips the increment
and the captures entirely, which is the point of putting it first.

The number that matters is the third column. At 100k events per second, a
counting watch costs about 5.5 ms of CPU per second — roughly 0.04% of one core
on this machine, and 0.003% of a 14-core node.

Note what `recent` and `enrich` cost on a telemetry source. Each is an ETS write
per event, and either roughly triples the handler. That is why both are opt-in,
and why the guide says to reach for them deliberately rather than by default. On
a **report** source, `enrich` is free — the report already contains the fields,
and the process is dead, so there is nothing to introspect anyway.

A watch with no filter and no captures compiles to a two-element tuple the
handler matches before incrementing, so the common case pays nothing for
features it did not ask for.

Note also that `:telemetry.execute/3` costs 40 ns before Kepler does anything.
If you are already emitting the event, Kepler is a fraction of a cost you have
already paid. If you are not, emitting it is the larger of the two decisions.

### Per tick, on Kepler's own process

50 tier 2 watches — the most expensive kind, since they sample every tick —
evaluate in **88 µs**, about 1.8 µs per watch. At the default 1-second tick on
a 14-scheduler node that is 0.0006% of the node's compute.

Telemetry watches are cheaper per tick than that unless they use percentiles,
which read a 960-slot counter array and difference it. That is still
microseconds.

### Memory

| | Bytes |
| --- | ---: |
| `:count`, `:rate`, `:sum` watch | ~1 KB |
| `:average` watch | ~2 KB |
| `percentile:` watch | ~115 KB |
| `recent size: N`, or `enrich` on telemetry | one ETS table each |

A percentile watch allocates 960 counter slots, and `:write_concurrency`
replicates them per scheduler — so the figure scales with your core count
(960 × 14 × 8 bytes here). This buys lock-free writes under concurrency, which
is the right trade for a hot path. If you declare dozens of percentile watches,
this is the line item to look at.

## The tier model

Every source belongs to a cost tier, and the declaration makes the tier
obvious. `Kepler.Watch` records it; `Kepler.watches/0` will tell you what you
actually declared.

### Tier 0 — free

Discrete sources that cost nothing until something happens.

`source system_monitor: {:long_gc, 500}` hands the VM a threshold and it
enforces it. Zero steady-state cost; a message arrives only when it trips. There
is no cheaper signal on the BEAM. The catch is that a node has exactly one
system monitor, so Kepler detects an existing one and yields rather than
stomping `:observer` or `:recon`.

`source crash_report: :any` and `source supervisor_report:` add a `:logger`
filter that pattern-matches the report label. If nothing in your system is
dying, that is a failed match per log event. Enrichment on these sources is free
— the report OTP assembled already contains the fields, and the process is dead,
so there is nothing to introspect even if you wanted to.

### Tier 1 — near-free

`source telemetry: [...]`. One atomic increment per event, on your process. The
handler is a captured named function with its counter reference passed in as
handler config, so there is no runtime lookup — `:telemetry`'s own ETS lookup
happens once per event regardless of how many watches you declared on it.

### Tier 2 — cheap, and constant

`source process:` and `source vm:`. A few reads per tick, independent of your
traffic. Watches on the same process are read in one `Process.info/2` call;
watches on the same VM group share one read.

Kepler **never** calls `:erlang.processes()`. On a node with half a million
processes that is ruinous at 1 Hz, and it is the single easiest way to turn a
lightweight observer into the thing you are debugging. Watch named things
precisely.

### There is no tier 3

Tracing is the highest-risk surface and the least necessary, so it is out of
scope. If you need it, `:recon_trace` already has the rate limiter and hard
message cap that make it safe — use it directly, in a bounded window, and turn
it off.

## Self-budgeting

A library that asks to be always-on owes you a number rather than an assurance.
Kepler measures its own cost every tick and reports it:

```elixir
iex> Kepler.status().budget
%{
  share: 0.0006,               # fraction of node compute the pass consumed
  limit: 0.01,                 # what you configured
  level: 0,                    # 0 = everything running
  reductions_per_second: 4120, # Kepler's own processes
  schedulers: 14
}
```

`share` is `tick duration / (window × schedulers)` — measured with a monotonic
clock around the evaluation pass, not modelled. `reductions_per_second` covers
Kepler's processes as a whole, read per-process so it disturbs nothing.

Neither includes the telemetry handlers, because those run on your processes
and are one atomic increment each. That is the point of the design: the cost
that scales with your traffic is the cost that is too small to measure.

Every event payload carries the same numbers under `"kepler"`, so the overhead
travels with the notification.

### Shedding

Going over budget for `:shed_after` consecutive ticks raises the shed level;
staying under for `:restore_after` lowers it again. The hysteresis matters —
shedding on a single slow tick would make Kepler flap alongside whatever made
the node slow.

| Level | Effect |
| --- | --- |
| 0 | Everything runs. |
| 1 | Tier 2 sampling runs on one tick in four. |
| 2 | Level 1, and the tick interval is multiplied by four, up to `:max_tick`. |

Tiers 0 and 1 are never shed. A system monitor costs nothing until it trips and
a telemetry counter costs nothing to leave incrementing — turning either off
would save nothing and lose data.

```elixir
config :kepler,
  budget: [
    share: 0.01,          # 1% of the node; :infinity disables shedding
    shed_after: 3,
    restore_after: 10,
    max_tick: :timer.seconds(30)
  ]
```

## Tuning

**Raise the tick before you remove watches.** Evaluation cost is per tick, so
`tick: 5_000` cuts it fivefold. The cost is resolution: `sustained:` windows and
rate calculations become coarser, and a spike shorter than a tick is invisible.

**Prefer `:count` and `:average` over percentiles** when a mean would answer
the question. A percentile watch costs 20 ns more per event and 115 KB.

**Set `keys:` on every `recent`, and keep `enrich` lists short on telemetry
sources.** Telemetry metadata can be large, and both the ETS write and the
webhook payload carry it.

**Watch named processes, not many processes.** A `process:` watch is cheap; a
hundred of them on a hundred targets is a hundred `Process.info/2` calls per
tick.

## Reproducing these numbers

```elixir
# bench.exs — run with `mix run bench.exs`
defmodule Bench do
  use Kepler

  watch :counting do
    source telemetry: [:bench, :call]
    measure :count
    fire when: value > 1_000_000_000
  end
end

Application.stop(:kepler)
{:ok, _sup} = Kepler.Supervisor.start_link(watches: Bench, tick: 3_600_000, sinks: [])

measure = fn event ->
  {us, _} =
    :timer.tc(fn ->
      Enum.each(1..500_000, fn _i ->
        :telemetry.execute(event, %{duration: 1_234_567}, %{status: :ok})
      end)
    end)

  Float.round(us * 1000 / 500_000, 1)
end

IO.puts("no handler: #{measure.([:bench, :none])} ns")
IO.puts("watched:    #{measure.([:bench, :call])} ns")
```

For the evaluation pass, declare N watches, call `Kepler.tick/0` in a loop, and
read `Kepler.status().budget.share`.

## What this does not cover

- **Delivery.** Webhook delivery is I/O in a supervised task off the poller. It
  does not affect the tick, and a wedged endpoint produces counted drops rather
  than back pressure. See
  [sinks and payloads](sinks_and_payloads.md#delivery-and-what-happens-when-it-fails).
- **Your telemetry events.** If emitting the event is expensive, Kepler cannot
  make it cheaper.
- **Clusters.** Each node runs its own Kepler and does no cross-node
  correlation. That is a deliberate limit, not an oversight: distributed
  coordination in the hot path is how lightweight things stop being
  lightweight.