Writing watches

The whole language, and what each part costs.

defmodule MyApp.Watches do
  use Kepler

  watch :name do
    source ...      # required — where occurrences come from
    measure ...     # sampled sources only — what to read and how to fold it
    filter ...      # optional — discard events before counting
    enrich ...      # optional — fields to carry into the event
    recent ...      # optional — keep the last N observations
    severity ...    # optional — how loudly to react, defaults to :warning
    sink ...        # optional — which sinks to route to, defaults to all
    meta ...        # optional — static fields for the consumer
    fire ...        # optional — when to say something
  end
end

use Kepler imports watch/2 and nothing else. The names inside the block are imported for the duration of the block and withdrawn afterwards, so filter and recent cannot collide with the rest of your module.

A watch with no fire clause on a sampled source is measurement-only: it computes a value every tick and other conditions can read it, but it never notifies anyone.

Naming

The watch name is the stable event id consumers route on. It is the public contract. Renaming a watch is a breaking change for whoever is on the other end of the webhook, so choose a name that describes the occurrence rather than the current threshold — :checkout_latency, not :checkout_over_2s.

Names must be unique across every module you configure. Kepler refuses to boot otherwise.

Sources

Exactly one per watch. They come in two shapes.

Discrete sources deliver occurrences. There is nothing to measure and nothing to wait for a tick to notice, so they use fire immediately.

Sampled sources produce a number every tick, and need a measure and a fire when: condition.

crash_report: and supervisor_report: — tier 0, discrete

source crash_report: :any
source supervisor_report: :child_terminated

The reason to be in-process. See crash attribution — it covers what these can honestly tell you, why :request_context depends on your application, and why you want config :logger, handle_sasl_reports: true.

system_monitor: — tier 0, discrete

source system_monitor: {:long_gc, 500}
source system_monitor: {:large_heap, 10_000_000}
source system_monitor: {:long_schedule, 200}
source system_monitor: :busy_port
source system_monitor: :busy_dist_port

You hand the VM a threshold and it enforces it. Nothing polls, nothing samples, and a message arrives only when the threshold is exceeded.

Several watches may declare the same type with different thresholds. Kepler installs the most sensitive one and routes each message to the watches it actually exceeded.

telemetry: — tier 1, sampled

source telemetry: [:my_app, :checkout, :stop]

Attaches a :telemetry handler. The handler runs inline in the process that emitted the event and does one atomic increment. Several watches on the same event name share a single handler.

process: and vm: — tier 2, sampled

source process: MyApp.ExportWorker
source vm: :memory      # :erlang.memory/0
source vm: :system      # counts, run queue, reductions
source vm: :scheduler   # utilization

Read once per tick. Watches on the same target or group are read together.

Kepler never calls :erlang.processes(). On a node with half a million processes that is ruinous at 1 Hz, and it is the easiest way to turn a lightweight observer into the thing you are debugging.

Measurements

On a telemetry source

measure :count                                   # events in the tick window
measure :rate                                    # events per second
measure :duration, :sum
measure :duration, :average
measure :duration, percentile: 99
measure :duration, percentile: 99, unit: {:native, :millisecond}
measure :duration, :average, unit: {:native, :millisecond}

unit: converts before recording, which matters more than it looks: telemetry durations are in :native units, so without it value > 2_000 compares against a number with no meaning you can reason about. Convert once, at the source, and your condition reads in the unit you wrote it in.

Percentiles come from a log-linear bucketed histogram — relative error bounded under 6.25%, and the reported value is the bucket's upper bound, so "p99 = 2431" means "99% of samples were at or below 2431". It never undershoots.

Values that are missing or not numbers are skipped rather than raised on, because a raising telemetry handler is detached by :telemetry permanently.

On a process source

:message_queue_len, :memory, :heap_size, :total_heap_size, :stack_size, :reductions, :alive.

:alive is 1 or 0, so "the exporter died" is a condition you can write:

watch :exporter_down do
  source process: MyApp.Exporter
  measure :alive
  severity :critical
  fire when: value == 0, sustained: :timer.seconds(10)
end

Every other measurement reports no data when the process is not running, so a watch on a restarting process holds its state rather than firing on the gap.

On a VM source

Two caveats, both documented in Kepler.Source.VM: reading :reductions resets the node's "reductions since last call" counter, and :utilization turns on the VM's scheduler_wall_time flag.

Unknown measurement keys are a compile error, with the available ones listed.

Conditions

fire when: value > 2_000
fire when: value > 2_000, sustained: :timer.seconds(30)
fire when: rate > 100, cooldown: :timer.minutes(5), resolve: true
fire immediately                                    # discrete sources only
fire immediately, cooldown: :timer.minutes(1)

The when: expression is captured unevaluated and compiled into a function on your module. There is no runtime parsing, no string evaluation, and no interpreter — the condition is ordinary compiled Elixir.

Names available in a condition

rate means the same thing everywhere: how fast is this number moving. If you want throughput, that is measure :rate, and then value is your events-per-second and rate is its acceleration.

On the first tick a watch has nothing to difference against, so prev equals value and delta is zero.

Referring to other watches

Correlating signals is what plain metrics alerting is bad at, and it is nearly free here because everything is already evaluated in one place on one tick.

watch :backlog_growing do
  source process: MyApp.ExportWorker
  measure :message_queue_len
  fire when: delta > 0
end

watch :p99_rising do
  source telemetry: [:my_app, :checkout, :stop]
  measure :duration, percentile: 99, unit: {:native, :millisecond}
  fire when: delta > 50
end

watch :degrading do
  source vm: :scheduler
  measure :utilization
  severity :critical
  fire when: backlog_growing and p99_rising and value > 0.8,
       sustained: :timer.minutes(2)
end

All measurements are sampled before any condition runs, and conditions run in dependency order, so met? and watch always see current-tick data regardless of declaration order. firing? deliberately reads the previous tick: firing state is the output of debouncing, and letting a condition depend on this tick's output would be circular.

A cycle between conditions is a compile error. So is referring to a measurement-only watch by bare name, and so is any unknown identifier — a typo in a condition fails the build rather than producing a watch that is silently always false:

** (Kepler.CompileError) lib/my_app/watches.ex:22: watch :degrading uses unknown
identifier `backlog_growng` in its condition; available here: `value`, `prev`,
`delta`, `rate`, `backlog_growing`, `degrading`, `p99_rising`

When a watch has no data

A tick where a percentile watch saw no events, or a process watch's target was not running, produces no opinion. Kepler does not evaluate the condition and the watch holds its state — a sustained window is neither advanced nor reset.

Feeding a condition a zero it did not observe is how a quiet period turns into a false resolution.

Debouncing

Every reactive system dies of alert storms, so debouncing is part of the declaration rather than something a consumer bolts on afterwards.

sustained: — the condition must hold on every tick for this long before the watch fires. A single tick where it does not hold resets the window.

cooldown: — the minimum gap between notices. It gates both a repeat while the condition is still true and a fresh episode after a flap, so a condition that oscillates cannot notify more than once per cooldown either way. This is the one to reach for on a crash watch, where a restart loop can produce hundreds of occurrences a second.

A watch is edge-triggered by default. With cooldown: 0 it notifies once when the condition becomes true and then stays quiet for as long as it holds. Setting a cooldown opts into repeating — a reminder at most that often.

resolve: true — also send an event when the condition stops being true, identical but with "state": "resolved". Only a watch that actually notified can resolve.

Severity

severity :critical

One of :info, :warning, :error, :critical. Defaults to :warning.

Unlike everything else about a watch, severity is part of the required core of the event — always present, never null, and sent as the kepler-severity header so a consumer can route without parsing the body.

Sinks

sink :siem, guarantee: :at_least_once
sink :ops

Routes this watch to named sinks. A watch that names none goes to all of them. Naming a sink that is not configured is a boot failure.

guarantee: is :best_effort (the default and the only implemented class) or :at_least_once, which is accepted, warns at boot, and behaves as best-effort. See sinks and payloads.

Enrichment

enrich [:stacktrace, :process_state, :request_context]   # a report source
enrich [:remote_ip, :user_id, :path]                     # a telemetry source

Names the fields to carry into the event's context.enriched.

On a report source these come out of the report OTP already assembled, so they are free — and naming a field no report can supply is a compile error. See crash attribution.

On a telemetry source these are metadata keys, taken off each event as it arrives. That costs one ETS write per event on your hot path, so name only what your consumer needs.

Enrichment does not apply to process: or vm: sources — a sampled number has nothing to enrich it from, and saying so is a compile error.

Recent observations

recent 25
recent size: 25, keys: [:user_id, :path]

Keeps the last N observations and attaches them to the event as context.recent. An event that says "the queue passed 10,000" is a number going up; one that also carries the last 25 things through the queue is actionable.

For a telemetry source this captures %{measurements:, metadata:, at:} as events arrive — one ETS write per event on your hot path. Set keys:; telemetry metadata can be large, and a webhook payload should not be.

For a gauge or discrete source there is no per-event moment, so recent is the sequence of tick samples or occurrences and costs nothing extra.

Static metadata

meta team: "payments",
     runbook: "https://runbooks.example/checkout-latency"

Copied verbatim into context.meta on every event this watch emits. This is how a generic consumer routes without Kepler knowing anything about it.

Cost, at a glance

Kepler.Watch records the tier, and Kepler.watches/0 will tell you what you actually declared.

See performance for the measured numbers.