# Operations

This guide covers runtime checks, backups, cleanup, and safe production operation.

## Startup checks

Production should set:

```sh
export KMS_SESSION_HASH_KEY=<high-entropy-secret>
export ERL_CRASH_DUMP_BYTES=0
```

If crash dumps are not disabled in production, KMS startup fails when `:require_crash_dump_disabled` is true.

For local RMK in production, set an explicit key path:

```sh
export KMS_RMK_LOCAL_KEY_PATH=/var/lib/kms/rmk.key
```

At startup, KMS runs `KMS.RMK.ensure_ready/1` for the configured local or remote
provider. Startup fails with a redacted error if the key file, credentials, key
reference, endpoint, or network access is unusable. The check defaults to a
five-second bound:

```elixir
config :kms, :rmk_readiness_timeout_ms, 5_000
```

Set this below the deployment platform's startup timeout. Provider error payloads
are not included in the startup exception.

Run migrations before serving traffic:

```sh
mix ecto.migrate
```

## Health checks

`KMS.API` exposes:

```text
GET /health
GET /health/live
GET /health/ready
```

Use `/health/live` for liveness and `/health/ready` for dependency readiness.

Readiness checks database access, bounded RMK access, and API-token configuration
when serving HTTP. RMK failures appear only as `"rmk": "error"`; credentials,
provider payloads, and key details are not returned. `/health/live` does not call
the RMK provider.

## Telemetry and metrics

KMS emits `:telemetry` events and exposes `KMS.Telemetry.metrics/0`.

Track at least:

- KMS operation latency/error rate;
- RMK provider latency/error rate;
- cache hit/miss;
- factor failures;
- remote client request latency/error rate;
- HTTP request latency/error rate;
- readiness status.

See [Observability](observability.md).

## Logging rules

Never log:

- plaintext;
- ciphertext payloads;
- bearer/API/session tokens;
- passwords/passphrases;
- factor codes or TOTP seeds;
- RMK key files or cloud credentials;
- private keys or fallback private keys;
- request/response bodies for KMS routes.

Configure proxies, APM, error reporters, and structured loggers to drop request bodies and authorization headers.

## Backups

### SQLite + local RMK

Back up:

- `$KMS_DATA_DIR/kms.sqlite3` or `KMS_DATABASE_PATH`;
- `-wal` and `-shm` sidecars when present;
- local RMK key file;
- fallback private keys from offline custody;
- application DB rows containing ciphertext.

Store RMK key backups with separate access controls from database backups when possible.

### PostgreSQL

Back up KMS PostgreSQL database with normal database backup tooling.

Also back up RMK material/credentials separately.

## Restore tests

Test restore before production and after major changes:

1. restore DB backup;
2. restore RMK material;
3. start KMS in isolated environment;
4. run `KMS.RMK.ensure_ready()`;
5. decrypt representative ciphertext with correct AAD;
6. verify sessions/factors/bindings only if they are in scope for restore.

## SQLite operations

- Use one writer process per SQLite database file.
- Keep WAL mode enabled.
- Put DB and sidecars on durable local disk.
- Avoid ephemeral container filesystems for production KMS state.
- Back up DB and sidecars consistently.
- Use PostgreSQL for multi-node writer deployments.

## RMK operations

Startup and `/health/ready` run the RMK check automatically. You can also run it
explicitly during deployment:

```elixir
:ok = KMS.RMK.ensure_ready()
```

External RMK providers may add latency/cost. Every readiness request reaches the
provider, so choose probe frequency accordingly. Tune RMK cache TTL:

```elixir
config :kms, :rmk_cache_ttl_ms, :timer.minutes(5)
```

Lower TTL reduces time plaintext key material stays cached. Higher TTL reduces provider calls.

See [Root master keys](rmk.md).

## Sessions cleanup

The KMS OTP application starts periodic expired-session cleanup.

Configure interval:

```elixir
config :kms, :session_cleanup_interval_ms, :timer.minutes(5)
```

Session lifetime knobs:

```elixir
config :kms, :session_default_ttl_ms, :timer.hours(1)
config :kms, :session_max_ttl_ms, :timer.hours(1)
config :kms, :session_idle_timeout_ms, :timer.hours(1)
config :kms, :session_max_lifetime_ms, :timer.hours(12)
```

## Remote KMS operations

For `KMS.API`:

- serve only on trusted private networks or behind TLS;
- set high-entropy API token;
- disable request-body logging at proxy and app layers;
- monitor `/health/ready`;
- rotate API token through normal secret-management process;
- keep RMK credentials only on KMS authority hosts.

See [Remote KMS](use_cases/remote.md).

## Upgrade operations

For normal upgrades:

1. read release notes;
2. back up DB and RMK material;
3. deploy code;
4. run migrations;
5. check `/health/ready`;
6. run representative encrypt/decrypt smoke tests.

See [Migrations and upgrades](migrations-upgrades.md).