Crash attribution

The headline source, and an honest account of what it can tell you.

A crash event carrying the dying process's state, its last message, its stacktrace, and the request that caused it is not assemblable from outside the BEAM. No scrape interval reconstructs it and no log line contains all of it. This is the entire reason to run an observer in-process.

watch :process_crash do
  source crash_report: :any
  enrich [:stacktrace, :process_state, :last_message, :reason, :request_context]
  severity :error
  fire immediately, cooldown: :timer.minutes(1)
end

{
  "watch": "process_crash",
  "severity": "error",
  "context": {
    "enriched": {
      "reason": "%RuntimeError{message: \"kaboom\"}",
      "stacktrace": [["MyApp.Exporter", "handle_cast", 2, {"file": "exporter.ex", "line": 41}]],
      "process_state": {"pending": 812, "batch": "b-993"},
      "last_message": ["$gen_cast", "flush"],
      "request_context": {"request_id": "req-42", "user_id": 7}
    }
  }
}

The process is already dead

This is the constraint everything else follows from.

By the time a report arrives, the process is gone — Process.info/2 on the pid returns nil. There is nothing to introspect. Enrichment reads what OTP captured at crash time, not live state, so what you can get depends entirely on what crashed.

The good news is that the richest case is also the common one: most things worth watching in an Elixir application are OTP behaviours, and their own terminate report is the one that carries state.

Turn on SASL reports

Elixir disables SASL reports by default, which filters proc_lib crash reports and every supervisor report out before any handler sees them.

# config/config.exs
config :logger, handle_sasl_reports: true

Kepler warns at boot when you declare a watch this affects, rather than letting it look broken:

[kepler] SASL reports are disabled, so supervisor reports will never arrive and
crash reports will only cover OTP behaviours, not plain proc_lib processes. Add
`config :logger, handle_sasl_reports: true` to see them.

:request_context is your problem, not Kepler's

This is the field worth understanding before you build on it.

A log event carries the Logger metadata of the process that logged it, so :request_context is whatever your application put in Logger.metadata/1 on the process that died, with :logger's own bookkeeping removed.

Which means attribution works only if you propagate context into spawned processes. Kepler can read Logger.metadata; it cannot make your application set it.

# In the request path — this is the easy half.
Logger.metadata(request_id: conn.assigns.request_id, user_id: current_user.id)

# In anything you spawn — this is the half people forget.
parent_metadata = Logger.metadata()

Task.Supervisor.start_child(MyApp.TaskSupervisor, fn ->
  Logger.metadata(parent_metadata)
  do_the_work()
end)

If your Tasks do not inherit request context today, :request_context will be empty and no amount of library will fix it. That is an application discipline problem. Find out which it is in your app before you design a workflow around the field — it changes how much this source is worth to you.

If you already run OpenTelemetry, its context propagation gives you the same thing through a different door; put the trace id into Logger.metadata and it rides along.

Deduplication

One GenServer crash under a supervisor produces three reports: the behaviour's own terminate report, a proc_lib crash report, and a supervisor child_terminated report.

Kepler fires a crash_report: watch on the first report for a given pid and suppresses the rest for two seconds. The first is also the richest — the behaviour's report is the one carrying state and last message.

supervisor_report: matches exactly one report kind and is never deduplicated, so a watch on supervisor churn still sees every restart. Declaring both watches gives you one crash event and one restart event, which is usually what you want:

watch :process_crash do
  source crash_report: :any
  enrich [:reason, :stacktrace, :process_state, :request_context]
  severity :error
  fire immediately
end

watch :child_restarting do
  source supervisor_report: :child_terminated
  enrich [:supervisor, :child_id, :reason]
  severity :warning
  fire immediately, cooldown: :timer.minutes(5)
end

Debounce, because crash loops exist

A process in a restart loop can crash hundreds of times a second. cooldown: is the difference between an event and an outage of your own making:

fire immediately, cooldown: :timer.minutes(1)

Kepler's egress is bounded regardless — the queue drops rather than grows, and Kepler.status().emitter.dropped counts what went. But a cooldown is the correct fix, and dropping is the backstop.

Kepler never reports crashes in its own processes, so a delivery failure cannot feed itself.

Available fields

enrich takes any of these. Omitting enrich entirely gives you everything the report happened to carry.

Fields the report had nothing for are absent from the payload rather than null, so a key's presence means the value is real.

Naming a field that no report can supply is a compile error:

** (Kepler.CompileError) lib/my_app/watches.ex:8: watch :process_crash cannot
enrich with :vibes; a report carries: :ancestors, :child_id, :initial_call,
:label, :last_message, :message_queue_len, :messages, :pid, :process_state,
:reason, :registered_name, :request_context, :stacktrace, :supervisor

Cost

A :logger filter that pattern-matches the report label, and nothing else until something crashes. If nothing in your system is dying, this costs a failed match per log event — tier 0, the same class as system_monitor:.

Testing it

Crash a real process and assert on the event. See testing.