KMS observability

Copy Markdown View Source

KMS core observability is vendor-neutral. The application emits :telemetry events and exposes KMS.Telemetry.metrics/0; operators choose the reporter/exporter.

Built in

Telemetry events

KMS emits these events with low-cardinality, redacted metadata:

EventMeasurementsMetadata
[:kms, :operation, :stop]duration native timeoperation, status, optional safe error
[:kms, :rmk, :operation, :stop]duration native timeoperation, provider, status, optional safe error
[:kms, :cache, :lookup]countnamespace, result
[:kms, :signing, :operation, :stop]duration native timeoperation, status, optional safe error
[:kms, :factor, :attempt]countresult
[:kms, :remote_client, :request, :stop]duration native timemethod, endpoint, status, optional http_status
[:kms, :api, :request, :stop]duration native timemethod, route, status

Security rule: event metadata must never include plaintext, ciphertext, bearer tokens, passwords, factor codes, private keys, or RMK material.

Metrics definitions

Use KMS.Telemetry.metrics/0 with any Telemetry.Metrics reporter:

children = [
  {TelemetryMetricsPrometheus, metrics: KMS.Telemetry.metrics()}
]

Health endpoints

KMS.API exposes:

  • GET /health cheap compatibility check
  • GET /health/live liveness check
  • GET /health/ready readiness check for dependencies

Readiness checks database access and API token presence from config :kms, :api.

Optional recommendations

Prometheus / PromEx

PromEx is a good optional choice for Prometheus and Grafana dashboards. Keep it in the host application, not as a hard KMS dependency.

Recommended shape:

  • keep PromEx in the host app supervision tree
  • enable BEAM, Ecto, and HTTP server plugins
  • add KMS.Telemetry.metrics() through a custom PromEx plugin or manual metrics hook
  • add Grafana panels for KMS operation latency, RMK latency, cache hit/miss, factor failures, and readiness

Protect /metrics; do not expose it publicly without auth or network controls.

OpenTelemetry tracing

For tracing, use OpenTelemetry packages in the host release:

  • opentelemetry_bandit or Plug/Phoenix instrumentation for HTTP server spans
  • opentelemetry_ecto for database spans
  • opentelemetry_finch for Req/Finch outbound HTTP spans

Keep custom spans around operation boundaries only. Do not put sensitive payloads in attributes.

Structured JSON logs

Use normal Elixir Logger in KMS and configure structured logging in the host app, for example logger_json.

Recommended metadata allowlist:

  • request_id
  • method
  • route
  • status
  • operation
  • error_kind

Never log request/response bodies or secret material.

Error reporting

Sentry or similar tools should be configured by deployers, not required by KMS.

Recommended setup:

  • configure DSN through environment
  • attach Logger handler in host app
  • scrub headers and metadata
  • drop request bodies for KMS routes

VM metrics

Add telemetry_poller in the host release for BEAM metrics:

  • memory
  • run queue
  • process count
  • port count
  • atom count

These complement KMS metrics but should not be required by the library.