Eventually consistent durable KV store with opt-in per-key linearizable CAS.

EKV supports three runtime roles:

• member mode — the default. Stores data on disk, replicates to other members, serves eventual reads locally, and participates in CAS quorum.
• observer mode — stores data on disk, replicates to other durable replicas, serves eventual reads and eventual writes locally, but routes CAS operations to voting members.
• client mode — stateless. Uses the same public API, but forwards operations to a selected voting member based on configured region preference.

In member and observer mode, EKV stores data on disk (survives restarts and node death) and replicates across all connected durable replicas automatically. There is no leader: every durable replica serves eventual reads and writes at all times, including during network partitions. CAS writes (if_vsn:, consistent: true, update/4) still require quorum and may fail when quorum is unavailable. Voting members execute CAS locally; observers route CAS to voting members and apply the committed result locally before replying on success. When connectivity is restored, durable replicas converge to the same state.

Quick Start

# In your supervision tree
{EKV, name: :my_kv, data_dir: "/var/data/ekv"}

# Basic operations
EKV.put(:my_kv, "user/1", %{name: "Alice"})
EKV.get(:my_kv, "user/1")          #=> %{name: "Alice"}
EKV.delete(:my_kv, "user/1")
EKV.get(:my_kv, "user/1")          #=> nil

# Prefix scans
EKV.scan(:my_kv, "user/") |> Enum.to_list()
#=> [{"user/1", %{name: "Alice"}, {ts, origin_node}}]

EKV.keys(:my_kv, "user/") |> Enum.to_list() #=> [{"user/1", {ts, origin_node}}]

# TTL — entry expires after 30 minutes
EKV.put(:my_kv, "session/abc", token, ttl: :timer.minutes(30))

# Subscribe to changes
EKV.subscribe(:my_kv, "rooms/")
EKV.put(:my_kv, "rooms/1", %{title: "Elixir"})
# receive
# => {:ekv, [%EKV.Event{type: :put, key: "rooms/1", value: %{title: ...}}], %{name: :my_kv}}

# CAS setup (requires cluster_size; node_id auto-generates if omitted)
{EKV,
 name: :my_kv_cas,
 data_dir: "/var/data/ekv_cas",
 cluster_size: 3,
 node_id: 1,
 wait_for_quorum: :timer.seconds(30)}

# Client mode — stateless EKV API that routes to voters by region order
{EKV,
 name: :my_kv_client,
 mode: :client,
 region: "ord",
 region_routing: ["iad", "dfw", "lhr"],
 wait_for_route: :timer.seconds(10),
 wait_for_quorum: :timer.seconds(10),
 shutdown_barrier: :timer.seconds(5)}

# Observer mode — local eventual reads/writes, CAS routed to voters
{EKV,
 name: :my_kv_observer,
 mode: :observer,
 data_dir: "/var/data/ekv_observer",
 cluster_size: 3,
 region: "lhr",
 region_routing: ["iad", "dfw", "lhr"],
 wait_for_route: :timer.seconds(10),
 wait_for_quorum: :timer.seconds(10)}

# CAS put via lookup + if_vsn
case EKV.lookup(:my_kv_cas, "lock/job-123") do
  nil ->
    EKV.put(:my_kv_cas, "lock/job-123", %{owner: "node-a"}, if_vsn: nil)

  {_value, vsn} ->
    EKV.put(:my_kv_cas, "lock/job-123", %{owner: "node-a"}, if_vsn: vsn)
end

# Handling ambiguous CAS result explicitly
case EKV.put(:my_kv_cas, "lock/job-123", %{owner: "node-a"}, if_vsn: nil) do
  {:ok, vsn} ->
    {:locked, vsn}

  {:error, :conflict} ->
    :already_locked

  {:error, :unconfirmed} ->
    # Resolve by reading through the CAS barrier
    EKV.get(:my_kv_cas, "lock/job-123", consistent: true)
end

# Or opt in to internal ambiguity resolution for this call
EKV.put(:my_kv_cas, "lock/job-123", %{owner: "node-a"},
  if_vsn: nil,
  resolve_unconfirmed: true
)

Values can be any Erlang/Elixir term. They are stored as :erlang.term_to_binary/1 internally and deserialized with :erlang.binary_to_term/1 on read. Note: Avoid storing structs or anonymous functions within values. See Value Serialization Caveats below for more information.

Consistency Guarantees

EKV provides eventual consistency: all nodes will converge to the same state given sufficient time and connectivity. It does not provide:

• Read-your-writes across nodes. A write on node A is not immediately visible on node B. Replication is asynchronous. On the local node, writes are immediately visible.

• Causal ordering. If node A writes key "x" then key "y", another node may see "y" before "x" (or "y" without "x" during a partition).

• Multi-key transactions. There is no way to atomically write multiple keys as a single unit. Per-key atomic read-modify-write is available via update/4 (CAS mode).

Conflict Resolution: Last-Writer-Wins (LWW)

For eventual-mode writes (put/4 and delete/3 without CAS options), EKV keeps the write with the highest nanosecond timestamp. If timestamps are identical (unlikely but possible with clock sync), the write from the lexicographically greater node name wins as a deterministic tiebreaker.

This means:

• The "latest" write always wins, where "latest" is determined by the writer's local wall-clock timestamp. EKV uses System.system_time(:nanosecond) as the base and ensures same-node local writes on a shard do not reuse a timestamp.

• Clock skew matters. If node A's clock is 5 seconds ahead of node B, then A's writes will beat B's writes for the same key even if B wrote "after" A in wall-clock time. For best results, run NTP or a similar time sync service on all nodes. Clock skew of a few milliseconds (typical for NTP) is fine — conflicts at that granularity are rare. Clock skew of seconds or more will cause surprising results.

• Deletes are writes too. A delete is a timestamped tombstone. If you delete a key on node A at time T1 and write the same key on node B at time T2 > T1, the write wins and the key reappears. This is by design — the higher timestamp always wins regardless of whether the operation was a put or a delete.

CAS-mode writes do not use LWW ordering; they are ordered by CASPaxos ballots.

Reads During Partitions

During a network partition, each side of the partition continues to serve reads and accept eventual writes independently. CAS writes continue only on partitions that can reach quorum; minority partitions return CAS errors such as {:error, :no_quorum} or {:error, :quorum_timeout}. When the partition heals, nodes sync and converge. For eventual writes, LWW determines the surviving value for conflicting writes on the same key.

Consistency Modes Per Key (Important)

EKV supports two write modes:

• Eventual/LWW mode — default put/4 and delete/3 without CAS options.
• CAS mode — put/4 with if_vsn: or consistent: true, delete/3 with if_vsn:, and update/4.

For predictable semantics, treat mode as key ownership:

• Different keys may use different modes in the same EKV instance.
• A key may start in eventual/LWW mode and later transition to CAS mode (LWW -> CAS is supported).
• Once a key is CAS-managed, eventual writes on that key are rejected (CAS -> LWW is not supported for writes).
• Keys managed via CAS should continue to use CAS write APIs.
• Reads for CAS-managed keys may be eventual (get/2, lookup/2) when staleness is acceptable, or consistent (get/3, consistent: true) when fresh linearizable reads are required.
• After transition to CAS mode, do not issue eventual writes on that key.

CAS Outcomes and :unconfirmed

CAS write APIs (put/4 with if_vsn: or consistent: true, delete/3 with if_vsn:, and update/4) can return:

• {:ok, ...} — write committed; returns the committed VSN.
• {:error, :conflict} — definite non-application for this attempt (for example stale if_vsn).
• {:error, :unconfirmed}: write entered accept phase, but the caller could not confirm final outcome. The write may already be committed, or it may have lost to a competing ballot and never committed.

On :unconfirmed, callers can issue get(name, key, consistent: true) to resolve current committed state before taking follow-up actions.

For convenience, pass resolve_unconfirmed: true on CAS writes to make EKV perform one internal barrier read and map ambiguous outcomes to current-state results: {:ok, ...} / {:error, :conflict} when resolvable, or {:error, :unavailable} if the resolution read cannot complete.

Configuration

All options are passed when starting EKV.

Member mode (default):

{EKV,
  name: :my_kv,
  data_dir: "/var/data/ekv",
  region: "iad",
  shards: 8,
  tombstone_ttl: :timer.hours(24 * 7),
  gc_interval: :timer.minutes(5),
  log: :info}

Observer mode:

{EKV,
  name: :my_kv_observer,
  mode: :observer,
  data_dir: "/var/data/ekv_observer",
  cluster_size: 3,
  region: "lhr",
  region_routing: ["iad", "dfw", "lhr"],
  log: :info}

Client mode:

{EKV,
  name: :my_kv_client,
  mode: :client,
  region: "ord",
  region_routing: ["iad", "dfw", "lhr"],
  log: false}

Transport Adapters

By default EKV sends member shard traffic and routed client RPC over Erlang distribution. :transport lets applications move that data plane onto an externally supervised volatile ordered lane:

{EKV,
  name: :my_kv,
  data_dir: "/var/data/ekv",
  transport: {MyApp.EKVTransport, name: :my_transport}}

The adapter contract is EKV.Transport. In {Module, opts}, EKV validates Module and passes opts directly to Module.init/1; option keys such as :name are adapter-owned and not interpreted by EKV. EKV initializes adapter state per replica shard process for member traffic, and initializes a lightweight adapter handle for each routed client RPC. Adapter init/1 must be cheap: validate opts, fetch/build a handle to an externally supervised transport, and return. It must not start per-call transport workers. EKV does not start or supervise the external transport.

Observer Mode

Observer mode is for nodes that should keep a full local durable replica and serve low-latency eventual reads locally, but should not expand the CAS voter set.

• Observers start SQLite, replication, GC, subscriptions, and anti-entropy.
• Eventual reads and eventual writes are local on the observer.
• CAS reads/writes route to a selected voting member based on :region_routing.
• Successful observer CAS calls apply the committed result locally before replying, so immediate local eventual read-your-CAS-writes is preserved.
• Observers do not count toward CAS quorum and clients do not route to them.
• {:error, :unconfirmed} on an observer means EKV could not confirm the committed state was safely visible for the caller's current path; resolve with get(name, key, consistent: true) before trusting local eventual state.

Client Mode

Client mode is intended for application nodes that need the EKV API but should not expand the durable replica set or CAS quorum size.

• Clients do not start SQLite, replication, GC, or blue-green machinery.
• Eventual reads are no longer local SQLite reads; they are routed to the selected voter.
• wait_for_route can delay startup until a voter route is selected.
• wait_for_quorum can additionally delay startup until that selected voter reports CAS quorum reachable.
• Members run periodic anti-entropy by default so a connected member that missed a prior update eventually replays the normal sync path and heals.
• scan/2 and keys/2 still return Elixir streams, but are backed by paged RPC to the selected voter.
• subscribe/2 works in client mode; client subscribers are delivered cluster-wide via :pg.
• If no backend is reachable, read APIs raise and write APIs return {:error, :unavailable}.
• After backend failover, eventual reads may observe an older replica view. Use consistent: true when freshness matters.

For deployment, scaling, blue-green rollout, quorum startup, and shutdown guidance, see OPERATORS.md in the repository.

Shutdown Barrier

shutdown_barrier: timeout_ms is an opt-in graceful-shutdown aid.

• In coordinated shutdowns, members can stay alive briefly while other members and clients enter terminal state, which reduces final-write :no_quorum failures and allows more replication to complete before exit.
• Blue-green outgoing members skip the barrier after successful handoff.
• This is best-effort only: crashes, :kill, and VM death bypass it.
• It does not replace correct supervision ordering; processes that must flush state should still shut down before EKV fully exits.

Startup Quorum Gate

CAS-enabled EKV instances can optionally block startup until quorum is reachable:

{EKV,
 name: :my_kv_cas,
 data_dir: "/var/data/ekv_cas",
 cluster_size: 3,
 node_id: 1,
 wait_for_quorum: :timer.seconds(30)}

This is useful when downstream children perform CAS reads or writes during their own init/1 or startup callbacks. With :wait_for_quorum set, EKV does not finish starting until quorum is reachable (or the timeout expires).

The gate is opt-in and only applies to CAS-configured instances. Eventual reads/writes never require quorum and are unaffected.

Choosing a Shard Count

The shard count controls write parallelism. Each shard is an independent SQLite database with its own writer process. Writes to different shards execute in parallel; writes to the same shard are serialized.

• 8 shards (default) — good for most workloads. Provides 8-way write parallelism, which saturates typical NVMe drives.

• 1–2 shards — appropriate for low-write-volume use cases (configuration stores, feature flags) where simplicity matters more than throughput.

• 16–32 shards — consider this for write-heavy workloads (>10k writes/sec) on fast storage, or when you have many cores and want to reduce lock contention.

• >32 shards — rarely needed. Each shard opens multiple SQLite connections and file descriptors. More shards means more memory and more files.

The shard count is permanent. It is persisted in each database file on first open. If you later start EKV with a different shard count against the same data_dir, it will raise an ArgumentError. To change the shard count, you must delete the existing data directory and start fresh. All replicas in the cluster must use the same shard count — a node with a mismatched count will have its replication connections rejected by members.

Tombstone Lifetime

When you delete a key, EKV doesn't immediately erase it. Instead, it writes a timestamped tombstone that replicates to all members, ensuring every node learns about the delete. Tombstones are permanently purged after tombstone_ttl (default 7 days).

If a node is offline for longer than the tombstone TTL, it may miss deletes that have already been purged from other nodes. EKV handles this with stale database detection: on startup, if the database's last activity timestamp is older than the tombstone TTL safety window, EKV refuses startup by default instead of trusting that on-disk state. This prevents "zombie" keys from reappearing. Operators can then either wipe that node's data dir so it rebuilds from members, or explicitly set allow_stale_startup: true when they intentionally want to trust the old on-disk cluster state.

Reduce tombstone_ttl if storage is tight and your nodes are rarely offline for long. Increase it if nodes may be offline for extended maintenance windows.

Long Live-Partition Protection

Startup stale-db detection covers nodes that were down and restarted. A separate edge case is a very long network partition where nodes stay up past tombstone_ttl.

By default (partition_ttl_policy: :quarantine), EKV detects reconnects after downtime longer than tombstone_ttl and quarantines that member pair instead of syncing potentially unsafe state. Replication remains blocked for that member until an operator rebuilds one side.

Down-since markers are persisted in kv_meta, keyed by node_id when available (fallback: node name). This preserves quarantine history across restarts and prevents node-name churn from bypassing quarantine when node_id is stable.

Anti-entropy also retries member_connect to current EKV.MemberPresence members missing from remote_shards. That lets transient false down-markers usually self-heal while they are still within tombstone_ttl.

Presence alone does not bypass overdue quarantine. If a persisted down-marker has already aged past tombstone_ttl, EKV still quarantines that reconnect by default until an operator rebuilds one side or explicitly widens the safety window.

Fallback name-based markers are bounded: EKV periodically prunes very old entries and caps retained fallback markers per shard.

Blue-Green Deployment

When deploying with a blue-green strategy on a single machine, two BEAM VMs run side by side briefly — the old release and the new one. If both VMs open the same SQLite files simultaneously, the result is corruption. The :blue_green option solves this with a synchronized handoff — the new VM coordinates with the old VM to drain operations, flush WAL, and close the writer before opening the same database files.

{EKV, name: :my_kv, data_dir: "/var/data/ekv", blue_green: true}

Startup behavior:

• First boot (no marker file) — writes the marker and operates normally.

• Same node restart (marker node matches node()) — no handoff needed. This is a normal EKV restart within the same VM.

• Different node (marker node does not match node()) — synchronized handoff. The new VM sends a handoff request to each shard on the old VM. The old VM drains pending CAS operations, persists its ballot counter, checkpoints the WAL, and closes its writer connection. Only then does the new VM open the database files. The old VM enters proxy mode, forwarding any remaining write requests to the new VM.

If the marker points at a dead or unreachable old VM, EKV skips handoff and opens the files directly. SQLite's WAL recovery handles any incomplete writes.

Requirements:

• Each deploy must use a different node name (e.g. include a timestamp or release version in the name). Both VMs must share the same node_id.

• The data_dir must be on a shared filesystem accessible to both VMs.

Disk space: No additional storage — both VMs use the same database files (sequentially, never simultaneously).

Multiple Instances

You can run multiple independent EKV instances in the same BEAM by giving each a different :name. Each instance has its own SQLite databases, its own replication mesh, and its own configuration. They do not interact.

children = [
  {EKV, name: :users, data_dir: "/data/users"},
  {EKV, name: :sessions, data_dir: "/data/sessions"}
]

Replication

Replication is automatic and requires no configuration beyond Erlang distribution. When a new node connects (via Node.connect/1 or a cluster manager like DNSCluster), EKV discovers member shards and syncs:

• Delta sync — if the nodes were recently connected and the replication log hasn't been truncated, only the missed entries are sent.

• Full sync — if this is the first connection or the node was away long enough for the oplog to be truncated, a full state transfer is performed (live entries + recent tombstones; expired rows are omitted).

After the initial sync, every local write is replicated to all connected members in real time (fire-and-forget, async). Consistency is maintained by LWW, not by delivery order.

TTL

Entries can be given a time-to-live:

EKV.put(:my_kv, "session/abc", token, ttl: :timer.minutes(30))

Expired entries are not returned by get/2, scan/2, or keys/2. Periodic GC emits :expired events for subscribers. For eventual/LWW keys, GC also writes a tombstone that replicates to other members. For CAS-managed keys, expiry stays local/lazy and long-expired rows are purged later instead of becoming replicated deletes.

Value Serialization Caveats

Because values are persisted to disk and may be deserialized by a different version of your application, avoid storing terms that are tied to the running code:

• Structs — a struct is a map with a __struct__ key pointing to a module atom. If the module is renamed, removed, or its fields change, deserialization will produce a bare map or a struct with missing/extra keys. Prefer plain maps (e.g. %{type: "user", name: "Alice"}) for durable storage.

• Anonymous functions — an anonymous function captures a reference to the module and function clause that created it. After a code deploy, that reference is invalid and :erlang.binary_to_term/1 will fail. Never store anonymous functions within values.

• PIDs, ports, references — these are ephemeral identifiers that are meaningless after a restart.

If you need to evolve your value schema over time, version-stamp your values:

# Write
EKV.put(:my_kv, "user/1", {2, %{name: "Alice", email: "a@example.com"}})

# Read with migration
case EKV.get(:my_kv, "user/1") do
  {2, data} -> data
  {1, data} -> Map.put(data, :email, nil)   # migrate v1 → v2
  nil       -> nil
end

This way old values written before a schema change are migrated on read without needing to backfill every key.