Normandy.Telemetry (normandy v2.2.0)

The catalogue of every :telemetry event Normandy emits.

This module has no functions. It is documentation, and test/telemetry/catalogue_test.exs keeps it honest: the test scans lib/ for emitted event names and fails when an event is emitted but not listed here, or listed here but not emitted. Each event below has a ### heading whose code span is parsed by that test (:a | :b as the last element means one event per alternative), so keep that shape when you add one.

Conventions that hold for all events:

• Durations and monotonic_time are in native time units; convert with System.convert_time_unit/3. Fields named *_ms say so.
• The keys Normandy adds never carry message, prompt or output content. The :exception reason is Normandy's own exception as raised or returned (an LLMError may embed provider or parse details); any other exception (a bug) is emitted as %Normandy.TurnError{type: :task_down, reason: %{exception: module}}, and a TurnError's non-Normandy :reason as %{exception: module}, because an exception's fields and message can hold the value it failed on. An exit reason or a thrown term loses every stacktrace in it and has such exceptions reduced at any depth of its tuples and lists; a call in it ({reason, {GenServer, :call, args}}) is emitted with its arity in place of its arguments, which hold the call's message; its other terms are emitted as they are. Keys with no value are left out of the map, except where a key is described as "always present".
• Metadata keys are snake_case atoms, not dotted semconv names. The mapping to OpenTelemetry GenAI attributes is in "OpenTelemetry GenAI mapping" below.
• Spans follow :telemetry.span/3 (through Normandy.Telemetry.Span): :start has system_time and monotonic_time measurements, :stop and :exception have duration and monotonic_time. :exception metadata adds kind (:error | :exit | :throw), reason and stacktrace, and each event of one span shares telemetry_span_context. The stacktrace of every Normandy :exception event is always []: a stacktrace can hold call arguments (the config, user data), so Normandy logs it and never emits it.
• The ids below appear on run, llm_call, request and tool events when known: turn_id (the memory turn id), session_id (set by Turn.Server) and conversation_id (the session id; there is no other conversation notion).

Spans

[:normandy, :agent, :run, :start | :stop | :exception]

One agent run: BaseAgent.run/3, run!/3 or stream/3, and each turn of Turn.Server (which emits these by hand; see "Turn.Server" below). It is the root of the agent's span tree: llm_call and tool spans run inside it.

• Start metadata: model, agent_name (always present, may be nil), provider (:anthropic, :openai_compatible, or client.options[:provider]; omitted when unknown), turn_id (only when known at start: Turn.Server knows it, BaseAgent admits input inside the span), session_id, conversation_id.
• Stop metadata: the start keys, plus turn_id from the finished run and the run totals (see "Run totals" below): llm_calls, input_tokens, output_tokens, cache_read_input_tokens, cache_write_input_tokens (each token total only when some call reported it), iterations, stop_reason and finish_reason; and answer_source (see "Answer source" below), which is not a total.
• Turn.Server only: stop and exception measurements add approval_wait (see "Approval wait"); start metadata carries resumed: true for a resumed durable turn.
• :exception from Turn.Server has kind: :error, reason: the exception run/2 replies, redacted as described above, and stacktrace: []. A turn that fails before it runs emits a run :start and :exception too: rejected input, a crash in admission and a crash in an after_tool hook while closing an interrupted batch (no turn_id), and a failed write of the turn's first messages (with the begun turn's turn_id).

A run :exception may be synthetic: see "Killed runs".

Run totals

A run's :stop sums the llm_call spans of that run that reached :stop:

• llm_calls - how many there were (always present).
• input_tokens, output_tokens, cache_read_input_tokens, cache_write_input_tokens - the sums of those calls' normalized counts, each only when some call reported it.
• iterations - the tool rounds the turn completed (a pause_turn continuation counts as one); 0 for an answer with no tool call.
• stop_reason - :completed, or :max_iterations when the answer came from the final call after max_tool_iterations rounds.
• finish_reason - the provider's stop reason of the last response, as a string ("end_turn"); absent when that response carries none (a run without tools gets its answer as the output schema, which is never read for one, even when it has a stop_reason field).

Not counted: a failed llm_call, compaction's summary calls (their own llm_call spans carry purpose: :summary; add them from those), and calls a custom call_llm handler (Turn.Server's or AgentProcess's :handlers) makes without an llm_call span, which gives llm_calls: 0. A resumed durable turn's run counts the calls made after the resume in llm_calls and the token totals, while iterations, stop_reason and finish_reason describe the whole turn (do not divide one by the other). A run that ends in :exception carries no totals.

Answer source

A run's :stop also carries answer_source (always present): :tool_round when a chat (:chat_message) agent's final response was blank and the answer is the text of this turn's last tool-calling response (see Normandy.Agents.BaseAgent.run/3), else :final. A run that ends in :exception has none.

[:normandy, :agent, :llm_call, :start | :stop | :exception]

One LLM call of a run (an iteration of the tool loop), covering all the provider requests behind it (retries, JSON parse retries, the structured-outputs fallback). Normandy.Context.Summarizer emits one per summary request too (compaction runs it inside the run); the run's own totals leave summaries out (see "Run totals"). BaseAgent.get_response/2 emits one, outside any run.

• Start metadata: model, iteration (1-based; absent on a summary call and on get_response/2), purpose (:summary, only on a summary call), agent_name, the ids, stream (boolean, always present), structured_output (boolean, always present), provider, max_tokens, temperature (only when set; Opus 4.7+ rejects it), effort, server_address (host of the client's base URL; absent when the client uses its library default).
• Stop metadata adds: has_tool_calls (boolean), tool_call_count (both absent on a summary call); usage, the raw provider usage map, exactly as reported (absent when the provider reported none, never nil); the normalized token counts input_tokens (for Anthropic it includes cache reads and writes; for OpenAI-shaped usage it equals prompt_tokens), output_tokens, cache_read_input_tokens, cache_write_input_tokens, each only when the provider reported it (see Normandy.Telemetry.Usage); response_id; finish_reason (a string, e.g. "end_turn", "stop"; absent when the provider sent none, as when an incremental guardrail halts the stream first); served_by (the model that answered, after a server-side fallback); input_transformations; attempts (provider requests made), json_retries, structured_fallback (boolean); time_to_first_chunk, streaming only.
• time_to_first_chunk is stop metadata, not a measurement, because :telemetry.span/3 fixes the stop measurements. It is in native units, measured from the start of the attempt that delivered the first chunk (retry backoff is not counted), and counts text deltas, thinking deltas and tool-use starts. It is absent when the stream delivered none.
• :exception: reason is the %Normandy.LLMError{}. A failed call still bills tokens, so read them from metadata.reason.usage (raw) and metadata.reason.usage_counts (normalized), nil when nothing was billed. kind is :error and stacktrace is [], for streaming calls too.

[:normandy, :llm, :request, :start | :stop | :exception]

One provider request inside an llm_call. A converse/3 can make several (the structured-outputs 400 fallback, JSON parse retries); BaseAgent's own retries are separate requests too. This is a different event from the llm_call span, and it is new (there has never been an [:normandy, :llm, :call] event).

• Emitted by the bundled adapters (ClaudioAdapter, OpenAICompatibleAdapter) for non-streaming calls and by BaseAgent for each stream attempt. A third-party Normandy.Agents.Model emits none unless it calls Normandy.Telemetry.Request.span/5 with the :request_tracker option it receives; llm_call's attempts then falls back to the number of converse/3 calls.
• Start metadata: model, provider, stream, attempt (1-based across the whole llm_call), request_kind (:initial | :retry | :json_retry | :structured_fallback; the key is not kind, which the exception event uses for the error class), the ids and agent_name when known.
• Stop metadata adds status: :ok, response_id, finish_reason (string) and this request's normalized token counts, each when known.
• :exception: kind: :error, reason is the %Normandy.LLMError{}, stacktrace is [] when the request returned the error instead of raising.

[:normandy, :tool, :execute, :start | :stop | :exception]

One tool execution, after the call passed policy, budget and hooks.

• Start metadata: tool_name, agent_name (always present), tool_type (:extension for an MCP-wrapped tool, :function otherwise), tool_call_id, tool_description (when the tool defines one; a raising tool_description/1 just leaves it out), the ids.

• Stop metadata adds status (:ok | :error) and, for a failed execution, error_type (:exception | :exit | :timeout | :tool_error | :invalid_result). The model sees the same {:error, message}; tool failures are returned, not raised, so they usually end in :stop with status: :error.

[:normandy, :memory, :command, :start | :stop | :exception]

One Normandy.Tools.Memory command, inside its [:normandy, :tool, :execute] span.

• Start metadata: command ("view" | "create" | "str_replace" | "insert" | "delete" | "rename", or nil when the model sent none) and store (the MemoryStore module). With the tool's telemetry_include_paths: true it adds namespace and path; both are off by default because they can carry personal data. On :start the namespace is the session namespace; on :stop it is the namespace the command actually used, which differs for a path inside a mount.

• Stop measurements add bytes_read, bytes_written and cas_retries.

• Stop metadata adds status (:ok | :error) and, on error, error_type (:not_found | :conflict_exhausted | :invalid_path | :read_only | :too_large | :no_match | :ambiguous_match | :invalid_line | :exists | :namespace_full | :backend | :invalid_input). Command errors are returned to the model, not raised, so they end in :stop.

[:normandy, :hook, :start | :stop | :exception]

One hook module call: one span per {module, opts} entry per event, for entries whose module exports the event's callback. No hooks, no spans.

• Start metadata: event (:on_session_start | :on_passivate | :on_prompt | :before_llm | :before_tool | :after_tool | :after_compact | :on_turn_end), hook (the module), agent_name, session_id, turn_id; before_tool and after_tool add tool_call_id and tool.

• Stop metadata adds result: :cont (passed the value on unchanged), :rewrite, :halt, :inject, :ok, or :rescued (a crash rescued under on_error: :open or :closed, or on a logged-only event).
• :exception fires when a crash propagates (on_error: :reraise, or :closed on after_compact/on_session_start, where it still propagates) or a return value breaks the callback's contract.

[:normandy, :workflow, :run, :start | :stop | :exception]

A multi-agent run: Workflow.execute/1, and the entry points of SequentialOrchestrator (execute/3, execute_with_messages/3), ParallelOrchestrator (execute/3) and HierarchicalCoordinator (execute/3, execute_with_tasks/3). Agent runs started inside nest under it, because the coordinators carry the OTel context into the tasks they spawn.

• Metadata: orchestrator (:workflow | :sequential | :parallel | :hierarchical), step_count, and for DSL workflows workflow_name (the module).

• Stop metadata adds status: :error when the entry point returned {:error, _}, :ok otherwise. A raise ends the span in :exception.
• Not wrapped: ParallelOrchestrator.execute_stream/2 (a lazy stream has no single extent) and the Reactive combinators. Their agent runs still nest under the caller's current span.

[:normandy, :a2a, :request, :start | :stop | :exception]

A2A.Server.handle_message/2,3,4 handling one agent-to-agent message.

• Start metadata: task_id, agent_name.

• Stop metadata adds status (:ok | :error) and, on error, error_type (:no_text, :too_large, :invalid_message or :agent_failed).

• Parenting: when the carrier passed to handle_message/3 has a valid W3C traceparent, the request span (and the agent run inside it) joins that trace. Only traceparent and tracestate are read; inbound baggage is dropped.

[:normandy, :turn, :approval, :start | :stop]

One park of a durable turn awaiting a tool-approval decision (Turn.Server only, hand-emitted; both events share telemetry_span_context).

• Start metadata: session_id, tool_names, count, plus the run's ids (turn_id, conversation_id).

• Stop metadata adds outcome: :approved | :rejected | :mixed | :expired | :terminated (:terminated when the server stops while parked). duration is the wait.

• There is no :exception event.

Events

[:normandy, :tool, :denied]

A tool call answered without running its tool. Measurement count: 1. Metadata: tool_name, agent_name, reason (:policy | :budget | :hook_halt | :approval_expired | :approval_rejected), tool_call_id, rule_id (the policy rule, when known), the ids. Emitted by Dispatch and by Turn.Server for rejected or expired approvals, so every denial has one shape.

[:normandy, :tool, :result_truncated]

A tool result longer than its cap, cut by Normandy.Agents.Dispatch before it reaches memory (error results too). Measurements: original_chars (the result's length, in characters: graphemes) and kept_chars (the cap; the note is added after them). Metadata: tool_name, agent_name, tool_call_id, the ids, and limit_source (:tool for a tool's Toolset.register/3 :max_result_chars, :agent for the agent's :max_tool_result_chars). Emitted where the call was dispatched, often a tool task, so match on tool_call_id or agent_name rather than the process.

[:normandy, :turn, :approval, :expired]

A parked batch whose approval window closed with no decision. Measurement count (parked calls). Metadata: session_id, mode (:continue | :fail), timeout_ms, tool_names, plus the run's ids (turn_id, conversation_id). Each call also gets a [:normandy, :tool, :denied], and the approval span stops with outcome: :expired.

[:normandy, :turn, :subscriber, :exception]

A Turn.Server :subscriber function raised, exited or threw; the turn continues. Measurement count: 1. Metadata: session_id, event (the subscriber event name), kind. This is a plain event, not a span.

[:normandy, :agent, :compaction]

Every run's compaction step, including a Compactor.NoOp. Measurements: whichever of tokens_before, tokens_after, tokens_removed the compactor reported. Metadata: agent_name, model, strategy, compacted (boolean), mid_round (boolean: false at the start of a turn, true after a tool batch, while the tool round is open), error, reason.

[:normandy, :agent, :guardrail, :violation]

One per guardrail violation set. Measurement count (violations). Metadata: stage, agent_name, guards (modules), violations, each reduced to %{guard:, path:, constraint:}: the guard, where and which rule failed, never the violation's message, so neither the offending text nor the guard's configuration (a forbidden term, a pattern) nor model text. The caller's ViolationError keeps the messages. Streaming adds streaming: true and mode.

[:normandy, :agent, :guardrail, :decision]

The admission decision of BaseAgent.admit/2,3, only when guardrails are configured. Measurement count. Metadata: stage (:input), outcome (:admit | :block | :error), agent_name, guards, violations (reduced as for :violation).

[:normandy, :turn, :unavailable]

Turn.Server could not load or start its session and is (still) :unavailable. Measurement count: 1. Metadata: session_id; phase: :start (the load after init/1, or the new session's on_session_start), :retry (the backoff timer's load failed again) or :request (a run/2 or an approval retried the load and it failed); reason, reduced as run/2's :session_unavailable reason is: atoms and integers kept, tuples walked, an exception its module ({Postgrex.Error, code} for one with a SQLSTATE), any other term a description of its type ("a map"). Never the reason's data.

[:normandy, :resume_reaper, :sweep]

One Normandy.Agents.Turn.ResumeReaper.reap/1 (on every :nodedown). Measurements: duration (native units), scanned (sessions list_resumable/1 returned), started, skipped (registered elsewhere, or their turn is no longer in flight) and failed (a registry lookup, a turn-state load or the start failed; each is logged). started + skipped + failed == scanned. Metadata: store (the SessionStore module), status (:ok, or :error when the listing failed, with every count 0) and, on :error, reason (reduced as for [:normandy, :turn, :unavailable]).

[:normandy, :agent_pool, :checkout]

One Normandy.Coordination.AgentPool.checkout/2 answered, emitted by the pool when it replies. Measurement wait: native units the caller waited for an agent to come back (0 when answered at once). Metadata: pool (the pool's pid), outcome (:ok | :timeout | :no_agents | :failed_to_start_agent), overflow (true when an overflow agent was started for this checkout). A blocked caller that dies before it is served emits nothing.

[:normandy, :json_deserializer, :fallback]

The JSON parse of a model response failed and a fallback ran. Measurement count: 1. Metadata: reason (the failure's tag), content_bytes. Never the output itself.

[:normandy, :circuit_breaker, :state_change]

A circuit breaker changed state. Measurements: count: 1, failure_count (failures counted in the state it left). Metadata: key (a shared breaker's Normandy.Resilience.CircuitBreaker.key/3: client module, endpoint without credentials or query, model and breaker options; nil for a breaker started with start_link/1), name (start_link/1's :name, else nil), from and to (:closed | :open | :half_open), reason (:failure_threshold | :timeout | :probe_success | :probe_failure | :probe_down | :reset | :trip). Emitted in the breaker's process, so outside the caller's trace context; the open-to-half-open change is emitted by the first admission (or state/1) after the open window.

[:normandy, :retry, :attempt]

A failed attempt that Normandy.Resilience.Retry will retry, emitted in the caller's process before the backoff sleep. Measurements: attempt (the attempt that failed, from 1), max_attempts, delay_ms. Metadata: error_type (an LLMError's type, an exception's module, :exit, :throw, :unexpected_return, an atom error itself, else :error). The last attempt, and an error that is not retried, emit nothing. LLM calls also emit one [:normandy, :llm, :request] span per attempt.

[:normandy, :session_registry, :ownership_lost]

A SessionRegistry.Redis refresh found a session key it held now holding another process, so two processes claimed one session; the owner stopped its local one with exit reason {:shutdown, :registration_lost}. Measurement count: 1. Metadata: session_id, registry (the owner's registered name). Emitted in the owner's process, outside any caller's trace context. A key that merely lapsed is claimed again and emits nothing.

Turn.Server

Turn.Server is a :gen_statem whose turn spans several callbacks, so it emits the run events by hand instead of through :telemetry.span/3.

• Parenting: run/2 carries the caller's OTel context in the call, and the server makes it current for the whole turn, from admission to on_turn_end, then puts back the context it had before. The run nests under the caller's span; llm_call and tool spans, which run in the server's tasks, nest under the run. The hook spans the server runs outside the run (admission's on_prompt, the after_tool of an interrupted batch's results, on_turn_end) are children of the caller, siblings of the run. BaseAgent admits input inside its run span, so there on_prompt is the run's child. A resumed durable turn uses the context saved with it the same way.
• Terminate: a server that stops with a turn in flight through a callback stop or :gen_statem.stop/3 emits a run :exception with kind: :exit in terminate/3. The server does not trap exits, so :kill, a supervisor's Process.exit/2 or a linked process dying run no callback and emit nothing.
• Resumed durable turns: a restarted server that resumes a turn emits a run :start with resumed: true, parented under the original caller's context, which was saved with the turn as a W3C carrier. It is a sibling of the original run in the same trace, not a child of it.
• approve/2 carries the approver's OTel context; it is current while the approval :stop and the denied events of rejected calls fire.

Approval wait

A turn that parks for approval is still one run span. The run's :stop and :exception measurements carry approval_wait: the sum of every park's duration in that turn (native units, 0 if never parked), so a consumer can separate waiting for a human from work. Each park is also its own [:normandy, :turn, :approval] span. BaseAgent.run/2 never parks, so its run events have no approval_wait.

Killed runs

Coordinators that kill an agent task (Reactive.race/some/all, ParallelOrchestrator.execute/execute_stream, Batch.Processor.process_batch) emit a synthetic [:normandy, :agent, :run, :exception] for it, in the coordinator process, with the caller's OTel context attached. It has reason: :killed, kind: :exit, stacktrace: [], no telemetry_span_context, and the same run metadata as a live run's :start (model, agent_name, provider, and the session ids when set), minus turn_id. Consequences:

• A handler can see this :exception with no matching :start (the start fired in the killed process).
• A process-keyed bridge such as opentelemetry_telemetry, which keeps its open spans in the process dictionary of the process that fired :start, cannot close that span from the coordinator, so the killed run's span is never exported. This is believed from that library's design; it is not verified (it is not a dependency here). Normandy's own test bridge, which works the same way, drops the event. A handler that wants the kill on the trace can record it from this event under the current (caller's) span.
• The llm_call and tool spans that were open inside the killed run stay open; nothing emits their :stop or :exception.
• Not emitted for agent pids, for tasks that finished their run before the kill, or for tasks Task.async_stream shuts down when the consumer halts early (Reactive.all with fail_fast). A short window between a run's :stop and its completion mark can still produce a late synthetic :exception.

OpenTelemetry GenAI mapping

Normandy depends on no OTel library (the packages are test-only). A bridge in your application turns these events into spans. The table says which Normandy key feeds which GenAI semantic-convention attribute. Checked against semantic-conventions-genai main on 2026-10-03 (memory operations on 2026-10-06). All GenAI attributes are Development stability, so names and allowed values can still change.

Gaps: Normandy has no gen_ai.agent.id, no gen_ai.request.top_p and the like, and never records message content (the opt-in content attributes are not supported). approval_wait has no standard attribute; export it as a custom attribute.

Example bridge

An illustrative sketch using opentelemetry_telemetry. It is not a dependency here, so this code is not compiled or run. It opens a span on :start and ends it on :stop or :exception:

defmodule MyApp.NormandyOtel do
  require OpenTelemetry.Tracer, as: Tracer

  @spans [
    [:normandy, :agent, :run],
    [:normandy, :agent, :llm_call],
    [:normandy, :tool, :execute]
  ]

  def attach do
    events = for span <- @spans, suffix <- [:start, :stop, :exception], do: span ++ [suffix]
    :telemetry.attach_many("my-app-normandy-otel", events, &__MODULE__.handle/4, nil)
  end

  def handle([:normandy | rest] = event, _measurements, meta, _config) do
    {span, [phase]} = Enum.split(rest, -1)

    case phase do
      :start ->
        OpentelemetryTelemetry.start_telemetry_span(
          :my_app,
          span_name(span, meta),
          meta,
          %{kind: kind(span), attributes: attributes(span, meta)}
        )

      :stop ->
        OpentelemetryTelemetry.set_current_telemetry_span(:my_app, meta)
        Tracer.set_attributes(stop_attributes(meta))
        OpentelemetryTelemetry.end_telemetry_span(:my_app, meta)

      :exception ->
        OpentelemetryTelemetry.set_current_telemetry_span(:my_app, meta)
        Tracer.set_status(:error, inspect(meta[:reason]))
        OpentelemetryTelemetry.end_telemetry_span(:my_app, meta)
    end

    event
  end

  defp kind([:agent, :llm_call]), do: :client
  defp kind(_span), do: :internal

  defp span_name([:agent, :run], %{agent_name: name}) when is_binary(name),
    do: "invoke_agent " <> name

  defp span_name([:agent, :run], _meta), do: "invoke_agent"
  defp span_name([:agent, :llm_call], meta), do: "chat " <> to_string(meta[:model])
  defp span_name([:tool, :execute], meta), do: "execute_tool " <> to_string(meta[:tool_name])

  # gen_ai.operation.name and gen_ai.provider.name are required by the conventions.
  defp attributes(span, meta) do
    %{
      "gen_ai.operation.name" => operation(span),
      "gen_ai.provider.name" => provider(meta[:provider]),
      "gen_ai.request.model" => meta[:model],
      "gen_ai.agent.name" => meta[:agent_name],
      "gen_ai.conversation.id" => meta[:conversation_id]
    }
    |> reject_nil()
  end

  defp stop_attributes(meta) do
    %{
      "gen_ai.usage.input_tokens" => meta[:input_tokens],
      "gen_ai.usage.output_tokens" => meta[:output_tokens]
    }
    |> reject_nil()
  end

  defp operation([:agent, :run]), do: "invoke_agent"
  defp operation([:agent, :llm_call]), do: "chat"
  defp operation([:tool, :execute]), do: "execute_tool"

  defp provider(:anthropic), do: "anthropic"
  defp provider(nil), do: nil
  defp provider(other), do: to_string(other)

  defp reject_nil(map), do: map |> Enum.reject(fn {_k, v} -> is_nil(v) end) |> Map.new()
end

A real bridge must also handle a run :exception that has no matching :start (see "Killed runs"); end_telemetry_span will find no open span and report an error.

Known limits

• Killed runs: see "Killed runs". Inner spans of a killed run stay open.
• Tool timeouts: a tool that runs past its timeout is shut down. Spans opened inside it (a memory command, a sub-agent's run, its llm_calls) never end; the tool span ends with status: :error, error_type: :timeout.
• Cross-process bridging: a process-keyed bridge ends a span only in the process that started it. Normandy carries the OTel context into the tasks it spawns (fan-out, Turn.Server tasks, AgentProcess), so parenting works across processes, but a span cannot be ended from another process.
• Parked runs: a run waiting for approval is one open span that can last minutes. Use approval_wait and the approval span to tell wait from work. A server killed while parked emits no run :exception, so that span is never exported.
• W3C propagation: A2A.AgentTool injects traceparent and tracestate into the A2A request, and A2A.Server.handle_message/3 extracts them from the carrier it is given. The server drops inbound baggage; AgentTool sends the host's own baggage when its propagator is configured to. The injection needs Claudio's A2A transport :headers option (Claudio 0.7.0). MCP calls carry no trace context: Claudio.MCP.Client.call_tool/4 has no metadata or header field, so an MCP server's spans start a new trace.
• Semantic conventions for GenAI are in Development status and may rename attributes.