# Encrypted

[![Hex.pm](https://img.shields.io/hexpm/v/ecto_encrypted.svg)](https://hex.pm/packages/ecto_encrypted)
[![HexDocs](https://img.shields.io/badge/hex-docs-blue.svg)](https://hexdocs.pm/ecto_encrypted)
[![CI](https://github.com/UntangleDev/ecto_encrypted/actions/workflows/ci.yml/badge.svg)](https://github.com/UntangleDev/ecto_encrypted/actions/workflows/ci.yml)
[![License](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/UntangleDev/ecto_encrypted/blob/main/LICENSE)

Authenticated, non-deterministic encryption for binary Ecto fields.

`Encrypted` is an `Ecto.ParameterizedType` backed by a PostgreSQL
`bytea` column. It encrypts every non-null value with AES-256-GCM, a fresh
96-bit IV, and a per-field key. Authentication binds the ciphertext to its
table and storage column. Moving a ciphertext to another table or column
therefore makes the load fail.

The package has one cipher and one envelope format. It uses OTP `:crypto` for
all cryptographic operations and has no third-party cryptography dependency.
It starts no process and stores no keys in ETS.

The package does not support deterministic encryption or encrypted queries.
Use `ecto_blind_search` beside an encrypted field when the application must
search it.

## Installation

Add `ecto_encrypted` to `mix.exs`:

```elixir
def deps do
  [
    {:ecto_encrypted, "~> 0.1"}
  ]
end
```

Generate a dedicated master key:

```elixir
Encrypted.generate_key()
```

The result contains 64 lowercase hexadecimal digits. A configured key may
instead contain the corresponding 32 raw bytes.

## Runtime configuration

Configure a map of key IDs to master-key sources and select one active ID:

```elixir
# config/runtime.exs
config :ecto_encrypted,
  keys: %{
    1 => {:system, "FIELD_ENCRYPTION_KEY_V1"},
    2 => {:system, "FIELD_ENCRYPTION_KEY_V2"}
  },
  active_key_id: 2
```

Key IDs are integers from 0 through 255. Encryption always uses the active ID.
Decryption reads the ID from the envelope and resolves that configured key.
Keep an old key configured until every ciphertext has been rotated away from
its ID.

Each key source may be:

- `:default`, which reads `config :ecto_encrypted, :master_key`;
- 32 raw bytes or exactly 64 hexadecimal digits;
- `{:system, "ENVIRONMENT_VARIABLE"}`;
- a zero-arity function; or
- an `{module, function, arguments}` tuple.

The type resolves keys when it encrypts or decrypts data. It does not resolve a
key while compiling the schema. A missing or malformed key raises
`Encrypted.Error`, and the error does not include key material.

Applications may declare `:keys` and `:active_key_id` on one field when that
field needs a separate ring:

```elixir
field :medical_note, Encrypted,
  keys: %{
    7 => {:system, "MEDICAL_NOTE_KEY"}
  },
  active_key_id: 7,
  redact: true
```

Prefer runtime application configuration for the common ring. A literal key in
a schema becomes part of compiled application code.

## Schema and migration

Store encrypted fields in nullable `bytea` columns:

```elixir
def change do
  alter table(:users) do
    add :email, :binary, null: true
  end
end
```

Declare the parameterized type and select its redaction behavior. Use
`redact: true` for the normal secure behavior:

```elixir
defmodule MyApp.User do
  use Ecto.Schema

  import Ecto.Changeset

  schema "users" do
    field :email, Encrypted, redact: true
    timestamps()
  end

  def changeset(user, attributes) do
    cast(user, attributes, [:email])
  end
end
```

Each encrypted field must set `redact: true` or `redact: false`. A schema-wide
redaction policy does not replace this field option. This rule gives the same
behavior across all supported Ecto versions. Schema compilation fails when the
field omits the option.

Ecto holds plaintext in the loaded struct and changeset. `redact: true` keeps
the field out of the schema's derived `Inspect` implementation, but it does not
redact application logs that interpolate the plaintext directly.
`redact: false` explicitly allows the plaintext in derived `Inspect` output.

`nil` passes through as `nil`. The type never encrypts `nil`. Empty binaries
are encrypted and produce a valid envelope with an empty ciphertext segment.
Non-binary values fail the Ecto cast.

Embedded schemas are not supported. Their JSON representation cannot carry
arbitrary envelope bytes without a separate binary-encoding contract. Declaring
an encrypted field inside `embedded_schema` raises while the schema compiles.
Calling `Ecto.Type.embedded_dump/3` for this type also raises. Store the encrypted
value in a database `:binary` field.

Every dump generates a new random IV. Dumping the same plaintext twice
therefore produces different bytes. Equality predicates over the encrypted
column cannot find a plaintext value.

## Compose with `ecto_blind_search`

Encryption and querying use separate columns and separate packages. The
changeset computes the blind index from plaintext before Ecto dumps the
encrypted field:

```elixir
defmodule MyApp.User do
  use Ecto.Schema
  use EctoBlindSearch

  import Ecto.Changeset

  schema "users" do
    field :email, Encrypted, redact: true

    blind_index :email,
      normalize: &__MODULE__.normalize_email/1

    timestamps()
  end

  def changeset(user, attributes) do
    user
    |> cast(attributes, [:email])
    |> validate_required([:email])
    |> put_blind_indexes()
    |> blind_unique_constraint(:email)
  end

  def normalize_email(email) do
    email
    |> String.trim()
    |> String.downcase()
    |> String.normalize(:nfc)
  end
end
```

The matching migration uses one `bytea` ciphertext column and one blind-index
column:

```elixir
def change do
  alter table(:users) do
    add :email, :binary, null: true
    add :email_bidx, :string, null: true
  end

  create unique_index(:users, [:email_bidx])
end
```

Query through the blind index:

```elixir
Repo.get_by(MyApp.User, MyApp.User.blind_index(:email, "person@example.com"))
```

Use different master keys for encryption and blind indexes. The two packages
also use different derivation domains, so a mistakenly shared master key does
not produce the same field key. Domain separation does not make intentional
key reuse acceptable. Separate master keys let operators rotate a leaked
blind-index key without re-encrypting data, or rotate an encryption key without
rebuilding indexes.

## Envelope format

Every stored value uses the v1 binary envelope:

```text
<<version::8, key_id::8, iv::binary-size(12),
  tag::binary-size(16), ciphertext::binary>>
```

| Offset | Size | Value |
|---:|---:|---|
| 0 | 1 byte | version, currently `1` |
| 1 | 1 byte | key ID, `0..255` |
| 2 | 12 bytes | random GCM IV |
| 14 | 16 bytes | GCM authentication tag |
| 30 | remaining bytes | ciphertext |

The version byte is always first. Any future change to the cipher, key-ID
width, tag, IV, derivation, or packing requires a new version. Unknown versions
fail; the loader does not guess.

The GCM associated data consists of the version and key-ID bytes followed by a
length-prefixed packing of:

```text
["ecto_encrypted/aad/v1", table, storage_field]
```

The header bytes are included so changing the key ID also fails
authentication. The packing starts with a little-endian 32-bit item count. Each
item then has a little-endian 64-bit byte length followed by its bytes.
Delimiter characters inside a table or field name therefore cannot make two
contexts ambiguous.

[`FORMAT.md`](FORMAT.md) defines the byte-level contract, key derivation, and
pinned regression vector.

## Table and column renames

The default context uses `schema.__schema__(:source)` and the field's database
source. Renaming either value changes the derived field key and associated
data. Existing ciphertext then fails authentication.

Preserve the old context during a rename:

```elixir
schema "renamed_users" do
  field :renamed_email, Encrypted,
    key_context: {"users", "email"},
    redact: true
end
```

`key_context` must name the table and storage column that originally produced
the ciphertext. It does not rename a database object.

## Rotate encryption keys

Rotation changes the active write key and re-encrypts existing rows in place.
Deploy in this order:

1. Add the new key ID to the runtime key ring. Keep every old key present.
2. Change `active_key_id` to the new ID and deploy that configuration to every
   writer.
3. Confirm that no old application instance can still write with the old
   active ID.
4. Re-encrypt the field:

   ```elixir
   Encrypted.Rotate.run(
     MyApp.Repo,
     MyApp.User,
     :email,
     batch_size: 100
   )
   ```

5. Run the same command again. A return value of `0` confirms that the scan
   found no row using an old key at that time.
6. Remove the old key only after every field and every deployment scope that
   used its ID has completed the second scan.

The runner requires one primary key. It reads rows in primary-key order and
uses one transaction per batch. Each update includes the primary key and the
old ciphertext bytes in its `WHERE` clause. If another writer changes the
field after the runner reads it, the bytes no longer match and the runner
leaves the newer value intact. The next run examines that row again.

Scope a run with a query that contains only `where` clauses:

```elixir
query = from(user in MyApp.User, where: user.tenant_id == ^tenant_id)

Encrypted.Rotate.run(
  MyApp.Repo,
  MyApp.User,
  :email,
  query: query,
  batch_size: 50,
  timeout: 30_000
)
```

The runner rejects joins, ordering, grouping, limits, offsets, explicit locks,
and other structural expressions because they can change keyset pagination or
the set of rows updated.

## Failure semantics

Authentication failures are exceptions, not absent values. The type never
turns tampering into `nil` and never skips a damaged row.

`Encrypted.Error.reason` identifies the failure:

| Reason | Meaning and action |
|---|---|
| `:authentication_failed` | The ciphertext, context, or key does not match. Restore trusted bytes and verify the configured context and key. |
| `:malformed_envelope` | The stored value is shorter than the v1 header. Restore the original ciphertext. |
| `{:unknown_version, version}` | The running code does not support the stored version. Deploy compatible code or restore a compatible value. |
| `{:unknown_key_id, id}` | The key ring does not contain the envelope's ID. Restore that key before loading or rotating the field. |
| `:invalid_key` | A resolved key is not 32 raw bytes or 64 hexadecimal digits. Correct the runtime secret. |

`ArgumentError` reports programmer misuse such as an unknown type option,
invalid key ID, unsupported rotation query shape, or invalid batch size.

Errors never include plaintext, key material, or a complete ciphertext. They
may include a key ID or environment-variable name because operators need those
identifiers to restore configuration.

## Threat model

The package protects plaintext when an attacker obtains a database copy but
not the master keys. AES-GCM also detects changes to the envelope, table
context, or column context before returning plaintext.

Context binding prevents a database writer from moving a ciphertext to another
table or column and obtaining a valid load. It does not bind the ciphertext to
a row identity. An attacker may move a ciphertext between rows in the same
column. The package also does not detect deletion or replay of an older valid
ciphertext in the same column.

The envelope exposes the plaintext length because the ciphertext has the same
length. It also exposes the format version and key ID. Database indexes,
access logs, queries, and row relationships can expose additional metadata
outside this type.

An attacker who obtains both the database and a master key can derive the
affected per-field keys and decrypt those fields. An attacker who controls the
running application can observe plaintext before encryption or after load.
Application authorization, host security, backups, and log handling remain
separate controls.

Random IVs make repeated plaintexts produce different ciphertexts. The package
does not provide deterministic equality, frequency-hiding guarantees beyond
that randomization, streaming encryption, or large-binary chunking.

Version 1 uses the random-IV construction from
[NIST SP 800-38D, Sections 8.2.2 and 8.3](https://nvlpubs.nist.gov/nistpubs/legacy/sp/nistspecialpublication800-38d.pdf).
NIST limits authenticated-encryption invocations to `2^32` for one derived
field key across all application instances. The package does not count
invocations. Rotate to new master-key material before one field reaches that
ceiling. Assigning a new key ID to the same master-key bytes does not reset it.
Per-field derivation gives each table and storage column an independent limit.

The package does not compress plaintext before encryption. Compression mixes
attacker-controlled and secret input into a length side channel, as seen in
CRIME-style attacks. Compress data only in a separate design that accounts for
that leakage and defines its own envelope version.

## Migrating from `cloak_ecto`

The v1 envelope cannot read a Cloak ciphertext. Migrate with two columns:

1. Add a nullable `bytea` column for `Encrypted`.
2. Deploy code that writes both the Cloak field and the new field.
3. Backfill in batches by loading plaintext through the existing Cloak type and
   writing that plaintext through `Encrypted`.
4. Compare row counts and application reads, then switch all reads to the new
   field.
5. Stop writing the Cloak column. Remove it only after rollback no longer
   depends on it.

`Encrypted.Rotate` cannot perform this migration because it only reads
this package's versioned envelope. A future adapter may add a read-only Cloak
format reader, but v1 does not contain one.

## Non-goals

Version 1 deliberately excludes:

- deterministic encryption and query rewriting;
- KMS integration and envelope encryption;
- streaming and chunked large-binary formats;
- compression;
- a cipher-selection option; and
- Cloak wire-format compatibility.

Use a key resolver function or MFA when an application retrieves key bytes from
an external secret service. The package itself does not own that service's
lifecycle, credentials, retries, or cache.

## Package name

The Hex package and OTP application use `ecto_encrypted`, which identifies the
Ecto integration. The public type uses `Encrypted`, which keeps schema field
declarations and error references short.