📊 Observability & Telemetry

Nous instruments every layer of agent execution with :telemetry events. You get structured visibility into agent runs, provider API calls, tool executions, the fallback chain, hooks, skills, and workflows — without changing a line of your agent code.

This guide covers attaching the built-in handler, the full event catalog (every name verified against source), writing custom handlers, and exporting Prometheus metrics through the bundled Nous.PromEx.Plugin.

Quick Reference

• Just want logs in dev? → Default Handler
• Need to wire into your own metrics? → Custom Handlers
• Exporting to Prometheus/Grafana? → Prometheus via PromEx
• What events fire and when? → Event Catalog

Default Handler

For local development and debugging, Nous ships a logging handler that attaches to the core agent, provider, tool, context, and callback events:

Nous.Telemetry.attach_default_handler()

It logs at sensible levels:

• Agent runs → info
• Provider requests and stream connections → debug
• Tool executions, iterations, context updates, callbacks → debug
• Tool retries / timeouts → warning
• Agent, provider, and stream exceptions → error

Provider error reasons are summarized (status + a capped body snippet, headers dropped) so a failing upstream cannot flood your logs or leak response context.

Detach it when you're done — for example in tests:

Nous.Telemetry.detach_default_handler()

The default handler attaches the core events only. Fallback, hook, skill, and workflow events have their own handlers or are meant for custom instrumentation — attach to them explicitly (see below).

Event Catalog

All durations are emitted in :native time units. Convert with System.convert_time_unit(duration, :native, :millisecond). The canonical source of truth is the moduledoc of Nous.Telemetry (and Nous.Workflow.Telemetry for workflow payloads).

Agent Events

Provider Events

These live under the [:nous, :provider, ...] namespace (there is no [:nous, :model, ...] event namespace — the PromEx metric names use model, but the underlying events are always provider).

Provider Streaming Events

Tool Events

Context & Callback Events

Fallback Events

Use these to alert on provider degradation. See the workflows and best-practices guides for fallback configuration.

Hook & Skill Events

Workflow Events

Full payloads are documented in Nous.Workflow.Telemetry.

Operational Events

Research observability: the research pipeline (Nous.Research) does not emit its own telemetry namespace. Because it runs ordinary agents internally, you observe it through the [:nous, :agent, ...] and [:nous, :provider, ...] events above.

Custom Handlers

Attach your own handler with :telemetry.attach/4 (single event) or :telemetry.attach_many/4 (many events). The handler receives event_name, measurements, metadata, config:

:telemetry.attach(
  "my-nous-handler",
  [:nous, :agent, :run, :stop],
  fn _event, measurements, metadata, _config ->
    MyApp.Metrics.track_agent_run(
      metadata.agent_name,
      measurements.duration,
      measurements.total_tokens
    )
  end,
  nil
)

Cost & token tracking

Token counts on [:nous, :agent, :run, :stop] make per-run cost tracking trivial. Aggregate into an Agent, ETS table, or your metrics backend:

defmodule MyApp.CostTracker do
  use Agent

  def start_link(_),
    do: Agent.start_link(fn -> %{total_tokens: 0, requests: 0} end, name: __MODULE__)

  def stats, do: Agent.get(__MODULE__, & &1)

  def attach do
    :telemetry.attach(
      "cost-tracker",
      [:nous, :agent, :run, :stop],
      &__MODULE__.handle/4,
      nil
    )
  end

  def handle([:nous, :agent, :run, :stop], measurements, _meta, _config) do
    Agent.update(__MODULE__, fn s ->
      %{s | total_tokens: s.total_tokens + measurements.total_tokens, requests: s.requests + 1}
    end)
  end
end

Keep telemetry handlers fast and crash-free. A raising handler is detached by :telemetry after one failure, silently dropping your metrics. Do heavy work (DB writes, HTTP calls) off the calling process — hand off to a Task or a queue rather than blocking the agent's hot path.

For a complete, runnable walkthrough — custom handler, the default handler, and a live cost tracker — see examples/advanced/telemetry.exs.

telemetry_metrics

If you already use :telemetry_metrics (e.g. with Phoenix LiveDashboard or a reporter), define metric specs over the event names above:

import Telemetry.Metrics

def metrics do
  [
    counter("nous.agent.run.start.count"),
    distribution("nous.agent.run.stop.duration", unit: {:native, :millisecond}),
    sum("nous.agent.run.stop.total_tokens"),
    counter("nous.tool.execute.stop.count", tags: [:tool_name]),
    counter("nous.tool.timeout.count", tags: [:tool_name])
  ]
end

This plugs into telemetry_metrics_prometheus, telemetry_metrics_statsd, or Phoenix LiveDashboard. For a turnkey Prometheus setup, prefer the PromEx plugin below.

Prometheus via PromEx

Nous bundles Nous.PromEx.Plugin, a PromEx plugin that turns the telemetry events into Prometheus metrics. The plugin module is only compiled when PromEx is present, so it is an opt-in dependency.

Setup

Add PromEx (and its :plug dependency) to your mix.exs:

{:prom_ex, "~> 1.11"},
{:plug, "~> 1.18"}  # required by PromEx

Register the plugin in your PromEx module:

defmodule MyApp.PromEx do
  use PromEx, otp_app: :my_app

  @impl true
  def plugins do
    [
      # ... other plugins
      {Nous.PromEx.Plugin, []}
    ]
  end
end

Options

Nous.PromEx.Plugin implements event_metrics/1 and accepts:

• :otp_app — OTP app name (defaults to the PromEx module's setting)
• :metric_prefix — custom metric prefix (defaults to [otp_app, :nous], or [:nous] when no otp_app)
• :duration_unit — :second, :millisecond (default), :microsecond, or :nanosecond; also selects the histogram bucket scheme

Exposed metric groups

The plugin builds three PromEx.MetricTypes.Event groups:

• :nous_agent_event_metrics — from [:nous, :agent, :run, :stop] and [:nous, :agent, :run, :exception]: run duration, total/input/output tokens, tool-call count, iteration count, and an exceptions counter. Tagged by agent_name, model_provider, model_name.
• :nous_model_event_metrics — from the [:nous, :provider, :request, ...] and [:nous, :provider, :stream, ...] events: request duration, token distributions, request exceptions, stream-connect duration, and stream exceptions. Tagged by provider, model_name (and has_tool_calls for duration).
• :nous_tool_event_metrics — from [:nous, :tool, :execute, :stop] and [:nous, :tool, :execute, :exception]: execution duration, attempt count, and an exceptions counter. Tagged by tool_name, success/will_retry.

Naming caveat: the metric group and metric names use the word model (e.g. nous_model_request_duration), but they are driven by the [:nous, :provider, ...] events. The two are intentionally different — do not try to attach to [:nous, :model, ...]; no such event exists.

Once exposed via PromEx's /metrics endpoint, scrape it with Prometheus and build Grafana dashboards over agent latency, token spend, and tool failure rates.

Related Guides

• Production Best Practices — monitoring, reliability, and deployment patterns
• Workflows — workflow/node telemetry in context
• Hooks — hook lifecycle and the events they emit
• Skills — skill activation/deactivation events
• Troubleshooting — diagnosing failures surfaced by exception events