Dual-backend adapter for mirrored writes, fallback reads, and phased cutovers.

This backend fronts two concrete backends:

• :primary - usually the current production backend
• :secondary - usually the destination backend

and lets reads/writes be directed independently while optionally mirroring and promoting data for an online cutover.

Core Model

Runtime behavior is driven by four switches:

• :read_preference (:primary | :secondary) - where normal reads come from

• :write_target (:primary | :secondary) - where authoritative writes go first

• :fallback_reads (boolean) - if a read misses on preferred backend, try the other
• :mirror_writes (boolean) - replicate writes/deletes to the non-authoritative backend

Additional safety knobs:

• :promote_on_fallback (boolean) - copy fallback read result into read-preferred backend

• :mirror_mode (:best_effort | :required) - mirror failures ignored vs surfaced

• :secondary_required (boolean) - whether ensure_ready/1 must verify secondary

Read Flow

Reads always start from read_preference.

get(key)
  |
  v
read_preference backend
  |-- hit -------------------------------> return object
  |
  |-- not_found & fallback_reads=true
         |
         v
      other backend
         |-- miss -----------------------> return miss/error
         |
         |-- hit
               |
               |-- promote_on_fallback=false -> return fallback object as-is
               |
               `-- promote_on_fallback=true
                      |
                      v
                   put into read_preference
                      |-- success/conflict-resolved -> return read_preference object
                      `-- transient failure          -> return {:error, {:promotion_failed, reason}}

Why promotion matters

ETags/CAS tokens are backend-local. Returning a token from fallback backend and then writing against read-preferred backend can conflict even when data matches. Promotion-on-fallback avoids that mismatch by returning tokens from the active read backend after promotion. If promotion fails, this backend now returns an error instead of the fallback object so callers do not accidentally CAS against the wrong backend token.

Write Flow

Writes (put_object, try_claim, update_object) and deletes start at write_target.

write/delete(key)
  |
  v
write_target backend (authoritative for that operation)
  |-- failure ---------------------------> return error
  |
  `-- success
        |
        |-- mirror_writes=false ---------> return success
        |
        `-- mirror_writes=true
               |
               v
            other backend (mirror)
               |-- success/not_found(delete) -> return success
               |
               `-- error
                     |-- mirror_mode=:best_effort -> return success
                     `-- mirror_mode=:required    -> return {:error, {:mirror_failed, reason}}

Notes:

• Mirrored put drops :etag from options intentionally. This avoids cross-backend CAS token coupling.
• For delete, a mirror-side :not_found is treated as success.

Source Of Truth

There is no single permanent source of truth baked into this adapter. Source of truth is operationally defined by configuration at a point in time:

• Read truth: read_preference
• Write truth: write_target

During a cutover, truth can intentionally shift phase by phase. mirror_writes + promote_on_fallback are the mechanisms that keep both sides converged while shifting those pointers.

Example Rollout

Configure the supervisor with the mirror backend while moving from object storage to EKV:

{DurableServer.Supervisor,
 name: MyDurableSup,
 prefix: "my_app/",
 backend:
   {DurableServer.Backends.MirrorStore,
    [
      primary: {DurableServer.Backends.ObjectStore, object_store_opts},
      secondary: {DurableServer.Backends.EKVStore, [name: :durable_ekv]},
      read_preference: :primary,
      write_target: :primary,
      mirror_writes: true,
      fallback_reads: true,
      promote_on_fallback: true
    ]}}

One possible rollout sequence:

Phase 1: Shadow
  read_preference=:primary, write_target=:primary
  mirror_writes=true, fallback_reads=true, promote_on_fallback=true

Phase 2: Backfill
  copy historical objects into the secondary backend

Phase 3: Combined Cutover
  read_preference=:secondary, write_target=:secondary
  keep mirror_writes=true for rollback safety, then disable

Phase 4: Finalize
  switch to single-backend config (e.g. pure EKV)

promote_on_fallback: true ensures fallback reads are copied into the active read backend so returned CAS etags remain backend-local and safe for subsequent lock updates.

A read-only cutover (read_preference=:secondary, write_target=:primary) is not safe for existing DurableServer restarts because etags/vsns are backend-local. Existing-object restart reads the stored body and etag/vsn from read_preference, but lock acquisition and follow-up CAS writes go to write_target.

API Semantics

• list_all_objects_stream/3 reads only from read_preference. It does not merge both backends.
• ensure_ready/1 always checks primary. Secondary readiness check is optional (secondary_required: true).