Raxol.Harness.StallDetector (Raxol v2.6.1)

View Source

The stall / doom-loop detector: a pure supervision instrument that tells the human "the agent you're watching has wedged" instead of letting a lying spinner run.

Design laws

  1. Pure detection policy. Feed it observations, get a verdict. No processes, no timers, no clocks in here -- the caller owns time and passes timestamps in (observe/2) or polls with its own clock (check/2). Identical inputs always produce identical outputs.

  2. Independent budget. The escalation budget (escalations_left) is this detector's own counter for quality alarms. It is completely separate from any transport/error retry counting a caller may also keep -- a flaky network and a wedged agent are different failure classes and must never share a budget.

  3. Never auto-recover. The detector takes no corrective action on the agent -- no interrupt, no resample, no retry. Its entire public surface is construction and observation; judgment over visible output belongs to the human it reports to.

  4. Hard graceful terminal. Each distinct :stalled/:looping alarm spends one unit of the escalation budget. Once the budget is spent, further alarms are returned with standing_by: true -- the verdict still tells the truth (class + current evidence), but is never presented as a fresh escalation again. Deterministic termination, no alert fatigue. A repeat of the same alarm never spends budget either (it is flagged standing_by as "already reported"). The budget never refills within a detector instance; callers that want per-turn budgets construct a detector per turn.

  5. Honesty floor. No verdict without evidence: every non-:ok verdict carries the reason, the structured detail, and a human-readable summary naming exactly what triggered it. A fresh or insufficient observation window is :ok -- never suspect-by-default. False alarms erode trust in the instrument.

Signals

  • Repetition -- the same tool called with byte-identical arguments @default_repetition_threshold times within the recent call window (:looping); the count just below that is :suspect.
  • Ping-pong -- two distinct calls alternating A,B,A,B for @default_ping_pong_threshold full cycles (:looping); one cycle short of that is :suspect. Cycle lengths 1 and 2 only, exact argument match -- deliberately simple.
  • No-progress elapse -- a working agent with no new events for longer than a threshold. The thresholds default to the status strip's shipped hung heuristic (Raxol.Harness.StatusStrip.default_warn_after_ms/0 / default_hung_after_ms/0) rather than a parallel set of constants: past warn is :suspect, past hung is :stalled.

When a call-pattern signal and a time signal fire simultaneously the most severe wins (:looping > :stalled > :suspect), with the pattern signal breaking ties -- its evidence names the concrete wedge, which is more actionable than a bare elapsed number.

The detector is always-on by construction: it has no enabled flag. Whether to consult it at all is the caller's decision, made simply by constructing one (or not).

Wiring

The one shipped consumer is the status strip: pass the latest verdict as the strip's optional :stall_verdict state key and a :stalled/:looping verdict renders as a highest-priority ALERT: notice naming the evidence. observation_from_event/1 maps harness fixture/journal events (the Raxol.Harness.Fixture.Event shape) to observations, converting microsecond event timestamps to the millisecond domain the thresholds live in -- the same µs -> ms convention Raxol.UI.Components.Harness.Block uses for durations.

Summary

Types

observation()

Observations are the detector's whole input vocabulary

t()

Functions

check(detector, now)

Polls the detector against the caller's clock (same millisecond domain the observations used), returning {verdict, detector}. This is where the no-progress signal can fire; an active call-pattern alarm also stays visible here (a clock tick must never clear a loop verdict). With no activity ever observed the window is insufficient: :ok, per the honesty floor.

new(opts \\ [])

Builds a detector. Options (all with documented defaults above): :repetition_threshold, :ping_pong_threshold, :warn_after_ms, :hung_after_ms, :escalation_budget.

observation_from_event(event)

Maps one harness event (a Raxol.Harness.Fixture.Event struct or an event-shaped map: atom top-level keys, string payload keys, microsecond ts) to an observation, or nil when the event is not an observation (meta family, or no usable timestamp). A completed tool_use item becomes a :tool_call; every other loop-family event is :progress.

observe(detector, arg)

Feeds one observation, returning {verdict, detector}. An observation IS activity, so the no-progress clock resets here; only the call-pattern signals can fire from observe/2.

Types

observation()

@type observation() ::
  {:tool_call, name :: term(), arguments :: term(), at_ms :: integer()}
  | {:progress, at_ms :: integer()}

Observations are the detector's whole input vocabulary:

  • {:tool_call, name, arguments, at_ms} -- the agent invoked a tool.
  • {:progress, at_ms} -- any other observable activity (deltas, results, messages). Progress feeds the no-progress clock but never resets the call window: tool results always interleave tool calls, so a loop must stay visible through them.

t()

@type t() :: %Raxol.Harness.StallDetector{
  calls: [{term(), term()}],
  escalations_left: non_neg_integer(),
  hung_after_ms: pos_integer(),
  last_activity_at: integer() | nil,
  last_alarm: tuple() | nil,
  max_history: pos_integer(),
  ping_pong_threshold: pos_integer(),
  repetition_threshold: pos_integer(),
  warn_after_ms: pos_integer()
}

Functions

check(detector, now)

@spec check(t(), integer()) :: {Raxol.Harness.StallDetector.Verdict.t(), t()}

Polls the detector against the caller's clock (same millisecond domain the observations used), returning {verdict, detector}. This is where the no-progress signal can fire; an active call-pattern alarm also stays visible here (a clock tick must never clear a loop verdict). With no activity ever observed the window is insufficient: :ok, per the honesty floor.

new(opts \\ [])

@spec new(keyword()) :: t()

Builds a detector. Options (all with documented defaults above): :repetition_threshold, :ping_pong_threshold, :warn_after_ms, :hung_after_ms, :escalation_budget.

observation_from_event(event)

@spec observation_from_event(map()) :: observation() | nil

Maps one harness event (a Raxol.Harness.Fixture.Event struct or an event-shaped map: atom top-level keys, string payload keys, microsecond ts) to an observation, or nil when the event is not an observation (meta family, or no usable timestamp). A completed tool_use item becomes a :tool_call; every other loop-family event is :progress.

observe(detector, arg)

@spec observe(t(), observation()) :: {Raxol.Harness.StallDetector.Verdict.t(), t()}

Feeds one observation, returning {verdict, detector}. An observation IS activity, so the no-progress clock resets here; only the call-pattern signals can fire from observe/2.