# KMS observability

KMS core observability is vendor-neutral. The application emits `:telemetry` events and exposes `KMS.Telemetry.metrics/0`; operators choose the reporter/exporter.

## Built in

### Telemetry events

KMS emits these events with low-cardinality, redacted metadata:

| Event | Measurements | Metadata |
| --- | --- | --- |
| `[:kms, :operation, :stop]` | `duration` native time | `operation`, `status`, optional safe `error` |
| `[:kms, :rmk, :operation, :stop]` | `duration` native time | `operation`, `provider`, `status`, optional safe `error` |
| `[:kms, :cache, :lookup]` | `count` | `namespace`, `result` |
| `[:kms, :signing, :operation, :stop]` | `duration` native time | `operation`, `status`, optional safe `error` |
| `[:kms, :factor, :attempt]` | `count` | `result` |
| `[:kms, :remote_client, :request, :stop]` | `duration` native time | `method`, `endpoint`, `status`, optional `http_status` |
| `[:kms, :api, :request, :stop]` | `duration` native time | `method`, `route`, `status` |

Security rule: event metadata must never include plaintext, ciphertext, bearer tokens, passwords, factor codes, private keys, or RMK material.

### Metrics definitions

Use `KMS.Telemetry.metrics/0` with any `Telemetry.Metrics` reporter:

```elixir
children = [
  {TelemetryMetricsPrometheus, metrics: KMS.Telemetry.metrics()}
]
```

### Health endpoints

`KMS.API` exposes:

- `GET /health` cheap compatibility check
- `GET /health/live` liveness check
- `GET /health/ready` readiness check for dependencies

Readiness checks database access and API token presence from `config :kms, :api`.

## Optional recommendations

### Prometheus / PromEx

PromEx is a good optional choice for Prometheus and Grafana dashboards. Keep it in the host application, not as a hard KMS dependency.

Recommended shape:

- keep PromEx in the host app supervision tree
- enable BEAM, Ecto, and HTTP server plugins
- add `KMS.Telemetry.metrics()` through a custom PromEx plugin or manual metrics hook
- add Grafana panels for KMS operation latency, RMK latency, cache hit/miss, factor failures, and readiness

Protect `/metrics`; do not expose it publicly without auth or network controls.

### OpenTelemetry tracing

For tracing, use OpenTelemetry packages in the host release:

- `opentelemetry_bandit` or Plug/Phoenix instrumentation for HTTP server spans
- `opentelemetry_ecto` for database spans
- `opentelemetry_finch` for Req/Finch outbound HTTP spans

Keep custom spans around operation boundaries only. Do not put sensitive payloads in attributes.

### Structured JSON logs

Use normal Elixir `Logger` in KMS and configure structured logging in the host app, for example `logger_json`.

Recommended metadata allowlist:

- `request_id`
- `method`
- `route`
- `status`
- `operation`
- `error_kind`

Never log request/response bodies or secret material.

### Error reporting

Sentry or similar tools should be configured by deployers, not required by KMS.

Recommended setup:

- configure DSN through environment
- attach Logger handler in host app
- scrub headers and metadata
- drop request bodies for KMS routes

### VM metrics

Add `telemetry_poller` in the host release for BEAM metrics:

- memory
- run queue
- process count
- port count
- atom count

These complement KMS metrics but should not be required by the library.