Telemetry integration for Cyclium.

Events

Episode lifecycle

[:cyclium, :episode, :started]      — %{episode_id, actor_id}
[:cyclium, :episode, :completed]    — %{episode_id, actor_id, output_count, finding_count}
[:cyclium, :episode, :failed]       — %{episode_id, actor_id, error_class}
[:cyclium, :episode, :blocked]      — %{episode_id, actor_id, conversation_id, reason}
[:cyclium, :episode, :resumed]      — %{episode_id, actor_id}
[:cyclium, :episode, :dropped]      — %{actor_id, expectation_id}
[:cyclium, :episode, :canceled]     — %{episode_id, actor_id, reason}
[:cyclium, :episode, :sampled_out]  — %{actor_id, expectation_id}

Step events

[:cyclium, :step, :tool_call]       — %{tool, action, episode_id, actor_id, conversation_id}
[:cyclium, :step, :synthesis]       — meas %{duration_ms, input_tokens, output_tokens, total_tokens}, meta %{episode_id, actor_id, conversation_id, model, usage_reported}
[:cyclium, :step, :observation]     — %{actor_id, episode_id}

usage_reported (boolean) distinguishes a genuine zero-token synthesis from a synthesizer that omitted its usage key — the latter also logs a warning when the result names a model, since the missing figure otherwise undercounts budgets and cost telemetry silently.

Checkpoint events

[:cyclium, :checkpoint, :save_failed] — %{count}, meta: %{episode_id, actor_id, expectation_id, phase, error_class}

Emitted when a checkpoint write could not be persisted (most often state that doesn't survive a JSON round-trip — see Cyclium.CheckpointSchema.json_plain?/1). The episode continues, but resume will fall back to a fresh init/2 instead of restoring progress, so a spike here is worth alerting on.

Actor events

[:cyclium, :actor, :event_received] — %{actor_id, event_type}
[:cyclium, :actor, :overflow]       — %{actor_id, policy, expectation_id}

:event_received fires only when the event is relevant to the actor — one of its expectations subscribes to it, or it is a lifecycle transition of the actor's own episode. Every actor is subscribed to the whole bus, so an actor that merely observes another actor's broadcast and drops it is not counted as having "received" it.

Output events

[:cyclium, :output, :delivered]     — %{type, dedupe_key, ref}
[:cyclium, :output, :failed]        — %{type, dedupe_key, reason}
[:cyclium, :output, :deduplicated]  — %{type, dedupe_key}

Finding events

[:cyclium, :finding, :raised]       — %{finding_key, actor_id, class}
[:cyclium, :finding, :cleared]      — %{finding_key, actor_id, class}
[:cyclium, :finding, :expired]      — %{count}
[:cyclium, :finding, :escalated]    — %{finding_key, class, from, to}

Finding sweep events

[:cyclium, :finding_sweep, :completed] — %{duration_ms, expired_count, escalated_count}, meta: %{node}
[:cyclium, :finding_sweep, :failed]    — %{duration_ms}, meta: %{node, reason}

Work claims events

[:cyclium, :work_claims, :acquired]     — %{count, duration_ms}, meta: %{dedupe_key, owner_node}
[:cyclium, :work_claims, :steal]        — %{count, duration_ms}, meta: %{dedupe_key, owner_node}
[:cyclium, :work_claims, :busy]         — %{count, duration_ms}, meta: %{dedupe_key, owner_node}
[:cyclium, :work_claims, :renewed]      — %{count}, meta: %{dedupe_key, owner_node}
[:cyclium, :work_claims, :renew_failed] — %{count}, meta: %{dedupe_key, owner_node}
[:cyclium, :work_claims, :completed]    — %{count}, meta: %{dedupe_key, owner_node}
[:cyclium, :work_claims, :failed]       — %{count}, meta: %{dedupe_key, owner_node}

Workflow events

[:cyclium, :workflow, :started]        — %{workflow_id, instance_id}
[:cyclium, :workflow, :step_started]   — %{workflow_id, instance_id, step_id, episode_id}
[:cyclium, :workflow, :step_completed] — %{workflow_id, instance_id, step_id}
[:cyclium, :workflow, :step_failed]    — %{workflow_id, instance_id, step_id}
[:cyclium, :workflow, :step_retried]   — %{workflow_id, instance_id, step_id, attempt}
[:cyclium, :workflow, :completed]      — %{workflow_id, instance_id}
[:cyclium, :workflow, :failed]         — %{workflow_id, instance_id, step_id}
[:cyclium, :workflow, :duplicate_blocked] — %{count}, meta: %{workflow_id, subject_value, owner_node}

Conversation events

[:cyclium, :conversation, :started]             — %{conversation_id, actor_id}
[:cyclium, :conversation, :claimed]              — %{conversation_id, principal_id}
[:cyclium, :conversation, :resolved]             — %{conversation_id, outcome}
[:cyclium, :conversation, :abandoned]            — %{conversation_id, reason}
[:cyclium, :conversation, :timed_out]            — %{conversation_id}
[:cyclium, :conversation, :awaiting_participant] — %{conversation_id, actor_id}

Background work spans

The events above are point-in-time signals. Several of Cyclium's background workers are self-scheduled — they run on their own timer (or as a detached startup task) rather than in response to a caller — so the work they do holds no caller context. A host application that traces its database work therefore sees each query such a worker runs as its own parentless root span, with nothing tying the queries of one tick together or recording how long the tick took.

To let a host hold a single trace span open across a whole tick, each of these workers wraps its work in :telemetry.span/3 via span/3, emitting a :start/:stop (or :exception) pair under one shared event prefix:

[:cyclium, :background_work, :start]     — meas %{system_time, monotonic_time},
                                           meta %{work, node, ...scope}
[:cyclium, :background_work, :stop]      — meas %{duration, monotonic_time},
                                           meta %{work, node, ...scope, result}
[:cyclium, :background_work, :exception] — meas %{duration, monotonic_time},
                                           meta %{work, node, ...scope, kind, reason, stacktrace}

work is a low-cardinality atom naming the worker (e.g. :trigger_requests_poll, :recovery_sweep, :finding_sweep, :work_claims_heartbeat); a consumer opens its trace span on :start, names it by work, and ends it on :stop, so the queries in between nest beneath it. result (when present) is a small map of low-cardinality numeric counts summarising the tick.

These are a different kind of event from those above — a paired span rather than a business signal — so they are deliberately not in events/0 and are listed separately by span_events/0. A logging consumer attaching to events/0 is unaffected by them; a tracing consumer attaches to span_events/0.

Usage

# Attach a simple logger for development
Cyclium.Telemetry.attach_default_logger()

# Or attach your own handler to specific events
:telemetry.attach("my-handler", [:cyclium, :output, :delivered], &MyHandler.handle/4, %{})