# Docket telemetry

Docket emits operational spans and durable domain facts below `[:docket, ...]`.
Span and pruner durations use native monotonic units; reporters convert them.
Counts and gauges are measurements. Use
`Docket.Telemetry.metric_metadata/2` for metric labels. Raw domain events may
contain identities for logs and traces, but those values must never become
labels. Claim tokens are never emitted.

## Operational catalog

| Event | Measurements | Bounded metric metadata |
| --- | --- | --- |
| `[:docket, :lifecycle, :transaction, :stop|:exception]` | `duration` | `operation`, `result` |
| `[:docket, :store, :operation, :stop|:exception]` | `duration` | `operation`, `result` |
| `[:docket, :checkpoint, :observer, :stop]` | `duration` | `checkpoint_type`, `result`, `durable_success` |
| `[:docket, :checkpoint, :observer, :failure]` | `count` | `checkpoint_type`, `result`, `durable_success` |
| `[:docket, :postgres, :dispatcher, :state]` | `concurrency`, `demand`, `in_flight`, `poll_active`, `poll_pending` | none |
| `[:docket, :postgres, :dispatcher, :poll]` | `duration`, `demand`, `concurrency`, `in_flight`, `leases`, `poisoned` | `claim_policy`, `result`, `source` |
| `[:docket, :postgres, :dispatcher, :launch]` | `duration` | `result` |
| `[:docket, :postgres, :dispatcher, :shutdown]` | `duration`, `vehicles_remaining` | `result` |
| `[:docket, :postgres, :notification]` | `count` | `result` |
| `[:docket, :postgres, :run_store, :claim]` | duration, candidate/selection ages and counts, `demand`, `leases`, `poisoned`, `steals`, `claim_attempts` | `preference`, `fallback`, `result` |
| `[:docket, :postgres, :claim_policy, :admission]` | `duration`, `demand`, `leases`, `poisoned`, `contentions` | `implementation`, `result`, `contention_phase` |
| `[:docket, :postgres, :claim, :attempt]` | `count`, `claim_attempts` | `result` |
| `[:docket, :postgres, :claim, :poisoned]` | `count` | `reason` |
| `[:docket, :postgres, :admission, :release]` | `count` | `reason` |
| `[:docket, :postgres, :claim, :operation]` | `duration`, optionally `matched` | `operation`, `result` |
| `[:docket, :postgres, :claim, :fence_lost]` | `count` | `stage`, `result` |
| `[:docket, :postgres, :graph_cache, :fetch]` | `duration` | `result` |
| `[:docket, :postgres, :graph, :fetch, :stop|:exception]` | `duration` | `result` |
| `[:docket, :postgres, :graph, :compile, :stop|:exception]` | `duration` | `result` |
| `[:docket, :postgres, :run_codec]` | `duration`, `bytes` | `operation`, `result` |
| `[:docket, :postgres, :store]` | `duration`, `attempted_rows`, `selected_rows`, `encoded_bytes` as applicable | `operation`, `result` |
| `[:docket, :node, :execution]` | `duration`, `attempt` | `result` |
| `[:docket, :lifecycle, :committed]` | `count`, `checkpoint_seq`, `step` | `checkpoint_type`, `disposition`, `result` |
| `[:docket, :postgres, :vehicle, :stop|:exception]` | `duration` | `result` |
| `[:docket, :postgres, :vehicle, :discard]` | `count` | `stage`, `result` |
| `[:docket, :postgres, :vehicle, :crash]` | `count`, `claim_held_ms`, `claim_attempt` | `result` |
| `[:docket, :postgres, :vehicle, :drain]` | `committed_moments`, `elapsed_ms`, `claim_held_ms`, `claim_attempt` | `outcome`, `budget` |
| `[:docket, :postgres, :pruner, :pass]` | deleted-row counts and `duration` | `result` |

Lifecycle and nested store spans share a raw `lifecycle_ref`; the metric
projection drops it. Committed facts and observer work start only after the
outer transaction succeeds. Stale fences and rollbacks therefore emit no
committed fact. Observer completion means only that the callback returned
after durable success, not that an external system durably accepted an effect.
Claim selection/attempt and store events are operational attempt facts and may
describe work later rolled back; only `[:docket, :lifecycle, :committed]` and
the domain events are durable-success facts.

Use Ecto repository telemetry for checkout, queue, and query duration. Active
vehicles are execution processes, not checked-out connections: slow node work
can raise `in_flight` without holding a database connection.

Claims fence durable state only. A stolen claim can execute node code and
external effects more than once even though only one moment commits. External
effects require their own stable idempotency scheme.

## Fair-rotation evidence boundary

The TenantFair sticky-admission engine keeps the generic ClaimPolicy and claim events in the
catalog above. The admission event reports total duration and a normalized
`contentions` count; `contention_phase: :policy_cursor` distinguishes bounded
singleton-cursor pressure without adding identity labels. That phase is set
only when the ring function catches contention while acquiring its singleton
policy/cursor authority; a later database lock timeout retains the public
`lock_contention` error but reports `contentions: 0` and
`contention_phase: :none` because its phase is not proven.

Detailed cursor, visit, disposition, outcome, and epoch evidence is available
only from the disabled-by-default raw trace result mode used by deterministic
tests and trusted inspection. Production passes trace off and filters those
rows before the unchanged public decoder. Tenant ID, raw `scope_key`, run or
graph identity, call token, cursor token, and claim token are forbidden as
ordinary metric labels. Aggregate telemetry cannot reconstruct the per-target
bounded-bypass proof; consult the [TenantFair correctness
evidence](architecture/docket-tenant-fair.md#correctness-evidence) for the
checked-in validation coverage.

Aggregate TenantFair outcome observations distinguish queued promotion,
admitted-ready reacquisition, and expired admitted steal. Cap/debt denial is a
raw inspection disposition, not a production aggregate: ordinary production
calls intentionally filter inspection rows before decoding.
Admission release counts use the bounded reasons `future`, `external`,
`terminal`, `signal`, `host_incompatible`, `poison`, and `abandon_backoff`;
they never carry tenant or run identity.
The public `Docket.inspect_claim_policy/2` facade supplies token-free `queued`,
`admitted_ready`, `admitted_claimed`, and `debt` counts for trusted operational
inspection after the host application authorizes the actor and owner scope.

The normative populations, units, formulas, exclusions, and Legacy control are
in the [TenantFair fair-rotation contract](architecture/docket-tenant-fair.md#fixed-window-fair-rotation-contract).
Timing and query-plan measurements in the [benchmark guide](benchmarks.md)
remain separate regression evidence and never replace that correctness oracle.

## Benchmark derivations

- Throughput: rate of `lifecycle.committed.count`; checkpoint and superstep
  advancement come from its `checkpoint_seq` and `step` measurements.
- End-to-end and database latency percentiles: histogram the native `duration`
  measurements on lifecycle, store, poll, launch, graph, codec, and node events.
- Active work and demand: last-value gauges from dispatcher `state.in_flight` and
  `state.demand`. Compare with Ecto checkout telemetry; active node work must not
  imply a checked-out connection.
- Claim health: rates and histograms for claim `leases`, `steals`, `poisoned`,
  `claim_attempts`, operation results, `matched`, fence-loss stage, and vehicle
  `claim_held_ms`.
- Cache behavior: cache hit ratio is `hit / (hit + miss + incompatible +
  generation_invalidated)` from graph-cache fetch results.
- Logical write amplification: sum store `attempted_rows` and `encoded_bytes`
  per committed lifecycle count, correlated by the raw `lifecycle_ref`; these
  describe requested work and may include rolled-back or idempotent attempts.
  Use Ecto query telemetry and database statistics when physical affected-row
  or WAL amplification is required. Reads use `selected_rows` and are excluded.
- Retry and park behavior: group committed lifecycle counts by `disposition`;
  `:retry`, `:timer`, and `:budget` are durable outcomes, not proposed work.
- Prune impact: compare deleted-row and cascade rates and pass latency with
  lifecycle throughput and store bytes before and after each pass.

The vehicle drain's `elapsed_ms` and `claim_held_ms` are explicit millisecond
business measurements. Fields named `duration` use native telemetry monotonic
units.