Operations

Copy Markdown View Source

This guide covers runtime checks, backups, cleanup, and safe production operation.

Startup checks

Production should set:

export KMS_SESSION_HASH_KEY=<high-entropy-secret>
export ERL_CRASH_DUMP_BYTES=0

If crash dumps are not disabled in production, KMS startup fails when :require_crash_dump_disabled is true.

For local RMK in production, set an explicit key path:

export KMS_RMK_LOCAL_KEY_PATH=/var/lib/kms/rmk.key

At startup, KMS runs KMS.RMK.ensure_ready/1 for the configured local or remote provider. Startup fails with a redacted error if the key file, credentials, key reference, endpoint, or network access is unusable. The check defaults to a five-second bound:

config :kms, :rmk_readiness_timeout_ms, 5_000

Set this below the deployment platform's startup timeout. Provider error payloads are not included in the startup exception.

Run migrations before serving traffic:

mix ecto.migrate

Health checks

KMS.API exposes:

GET /health
GET /health/live
GET /health/ready

Use /health/live for liveness and /health/ready for dependency readiness.

Readiness checks database access, bounded RMK access, and API-token configuration when serving HTTP. RMK failures appear only as "rmk": "error"; credentials, provider payloads, and key details are not returned. /health/live does not call the RMK provider.

Telemetry and metrics

KMS emits :telemetry events and exposes KMS.Telemetry.metrics/0.

Track at least:

  • KMS operation latency/error rate;
  • RMK provider latency/error rate;
  • cache hit/miss;
  • factor failures;
  • remote client request latency/error rate;
  • HTTP request latency/error rate;
  • readiness status.

See Observability.

Logging rules

Never log:

  • plaintext;
  • ciphertext payloads;
  • bearer/API/session tokens;
  • passwords/passphrases;
  • factor codes or TOTP seeds;
  • RMK key files or cloud credentials;
  • private keys or fallback private keys;
  • request/response bodies for KMS routes.

Configure proxies, APM, error reporters, and structured loggers to drop request bodies and authorization headers.

Backups

SQLite + local RMK

Back up:

  • $KMS_DATA_DIR/kms.sqlite3 or KMS_DATABASE_PATH;
  • -wal and -shm sidecars when present;
  • local RMK key file;
  • fallback private keys from offline custody;
  • application DB rows containing ciphertext.

Store RMK key backups with separate access controls from database backups when possible.

PostgreSQL

Back up KMS PostgreSQL database with normal database backup tooling.

Also back up RMK material/credentials separately.

Restore tests

Test restore before production and after major changes:

  1. restore DB backup;
  2. restore RMK material;
  3. start KMS in isolated environment;
  4. run KMS.RMK.ensure_ready();
  5. decrypt representative ciphertext with correct AAD;
  6. verify sessions/factors/bindings only if they are in scope for restore.

SQLite operations

  • Use one writer process per SQLite database file.
  • Keep WAL mode enabled.
  • Put DB and sidecars on durable local disk.
  • Avoid ephemeral container filesystems for production KMS state.
  • Back up DB and sidecars consistently.
  • Use PostgreSQL for multi-node writer deployments.

RMK operations

Startup and /health/ready run the RMK check automatically. You can also run it explicitly during deployment:

:ok = KMS.RMK.ensure_ready()

External RMK providers may add latency/cost. Every readiness request reaches the provider, so choose probe frequency accordingly. Tune RMK cache TTL:

config :kms, :rmk_cache_ttl_ms, :timer.minutes(5)

Lower TTL reduces time plaintext key material stays cached. Higher TTL reduces provider calls.

See Root master keys.

Sessions cleanup

The KMS OTP application starts periodic expired-session cleanup.

Configure interval:

config :kms, :session_cleanup_interval_ms, :timer.minutes(5)

Session lifetime knobs:

config :kms, :session_default_ttl_ms, :timer.hours(1)
config :kms, :session_max_ttl_ms, :timer.hours(1)
config :kms, :session_idle_timeout_ms, :timer.hours(1)
config :kms, :session_max_lifetime_ms, :timer.hours(12)

Remote KMS operations

For KMS.API:

  • serve only on trusted private networks or behind TLS;
  • set high-entropy API token;
  • disable request-body logging at proxy and app layers;
  • monitor /health/ready;
  • rotate API token through normal secret-management process;
  • keep RMK credentials only on KMS authority hosts.

See Remote KMS.

Upgrade operations

For normal upgrades:

  1. read release notes;
  2. back up DB and RMK material;
  3. deploy code;
  4. run migrations;
  5. check /health/ready;
  6. run representative encrypt/decrypt smoke tests.

See Migrations and upgrades.