dgen_mem (DGen v0.4.2)

Copy Markdown View Source

A pure-Erlang, in-memory dgen_backend.

Implements the FoundationDB semantics the registry actually depends on — MVCC read versions, read-conflict detection, versionstamps, watches — with no NIF, no server, and no I/O. Two things follow:

  • The suite runs without FoundationDB installed. Worth having on its own.
  • Commit timing stops being a source of nondeterminism, which is the last one outside eta_sched and eta_time. Commits happen synchronously, in the calling process, in a deterministic order.

Fidelity, and where it stops

Key layout is exactly FDB's: erlfdb_subspace and erlfdb_tuple are pure Erlang, so this reuses them rather than reimplementing order-preserving tuple encoding, where a subtle error would surface as ordering bugs in dgen_queue.

What is modelled faithfully:

  • Read versions. A read sees the newest value written at or before the transaction's read version. Pinning a read version (set_read_version/2) really does read the past, which is what makes the registry's fencing test meaningful.
  • Read conflicts. A commit aborts with not_committed (1020) if any key in a read-conflict range was written after the transaction's read version. This is the mechanism the leader fence rests on (§5.1), so it has to be real.
  • Versionstamps. set_versionstamped_value/3 substitutes a 10-byte stamp — 8-byte commit version, 2-byte in-transaction batch order — so the registry's version key decodes and orders exactly as it does against FDB.
  • Watches. A watch fires when its key is written by a later commit.

What is not:

  • No storage-level partitioning, no coordinators, no real distribution. A simulation runs in one VM, so the database is a single serialisation point rather than a cluster that can partition beneath the system under test.
  • Write conflicts are not tracked. FDB conflicts a commit whose write set overlaps another transaction's read set. Here commits are serialised, so the losing transaction is simply the one that commits second, and it sees the other's writes. Nothing in dgen depends on the distinction.
  • No transaction size limit (transaction_too_large). Nothing in dgen approaches FoundationDB's 10MB transaction cap.
  • No retry backoff. FoundationDB backs off exponentially with jitter; transactional/2 here retries at once, up to 100 attempts. A timer:sleep/1 would be wrong under a virtual clock — a sleeper is neither runnable nor blocked on a receive, so eta_sched can never step it.

The MVCC window is modelled. Versions advance at FoundationDB's rate (VERSIONS_PER_SECOND = 1e6) and the window is the same version distance (MAX_READ_TRANSACTION_LIFE_VERSIONS, 5 seconds' worth), so a read older than the window raises transaction_too_old as it would against the real thing — and under a virtual clock those five seconds are reachable instantly.

The version rate follows whatever clock eta_time exposes, so start the clock before opening the database; doing it part-way through shifts the origin the rate is measured from.

Fault injection

set_faults/2 makes the failures the registry is built to survive reproducible: conflict_p (spurious not_committed, which a correct caller retries) and commit_fail_p (a non-retryable error). Both draw from a seeded RNG, so a run replays.

Determinism

State lives in ETS keyed by the database handle and nothing runs in a process of its own, so no call into this module can read as quiescence to eta_sched. Commit versions come from a counter, not a clock.

This documentation is LLM-generated. See the AI disclosure in README.md.

Summary

Functions

Deletes a database and everything in it. Safe when it does not exist.

The transaction's versionstamp, as a future that resolves after it commits.

Resets a transaction for retry when Code is retryable, mirroring erlfdb:on_error/2: the returned future succeeds for a retryable error and raises otherwise. A reset drops staged mutations and conflict ranges, cancels the transaction's watches, and takes a fresh read version.

Opens (or resets) a named in-memory database.

Clears all data, keeping the database open. Versions keep advancing.

Opens a database and a root directory, mirroring dgen_erlfdb:sandbox_open/2 so DGen.Case can swap backends without knowing which one it has.

Sets the commit fault policy.

Commit count, conflict count, current version, and key count.

Runs Fun in a transaction, retrying on retryable errors — the same contract as erlfdb:transactional/2. A transaction handle passed in is used directly, so nesting composes.

Watches Key, firing {Ref, ready} at the watcher when its value changes.

Types

db()

-opaque db()

fault_opts()

-type fault_opts() :: #{conflict_p => float(), commit_fail_p => float()}.

tx()

-type tx() :: {dgen_mem_tx, atom(), reference()}.

Functions

add(Tx, Key, Value)

-spec add(tx(), dgen_backend:key(), integer()) -> ok.

add_read_conflict_key(Tx, Key)

-spec add_read_conflict_key(tx(), dgen_backend:key()) -> ok.

clear_range(Tx, StartKey, EndKey)

-spec clear_range(tx(), dgen_backend:key(), dgen_backend:key()) -> ok.

close/1

-spec close(db()) -> ok.

Deletes a database and everything in it. Safe when it does not exist.

commit/1

-spec commit(tx()) -> dgen_backend:future().

create_transaction/1

-spec create_transaction(db()) -> tx().

dir_create(Db, Dir, Name)

-spec dir_create(db(), dgen_backend:dir(), term()) -> dgen_backend:dir().

dir_pack(Dir, TupleKey)

dir_pack_vs(Dir, TupleKey)

dir_range(Dir, TupleKey)

dir_remove/3

-spec dir_remove(db(), dgen_backend:dir(), term()) -> ok.

dir_unpack(Dir, PackedKey)

error_code/1

-spec error_code(term()) -> integer() | error.

get/2

-spec get(tx(), dgen_backend:key()) -> dgen_backend:future().

get_committed_version(Tx)

-spec get_committed_version(tx()) -> integer().

get_next_tx_id(Tx)

-spec get_next_tx_id(tx()) -> non_neg_integer().

get_range/4

-spec get_range(tx(), dgen_backend:key(), dgen_backend:key(), list()) ->
                   [{dgen_backend:key(), binary()}].

get_read_version(Tx)

-spec get_read_version(tx()) -> dgen_backend:future().

get_versionstamp(Tx)

-spec get_versionstamp(tx()) -> dgen_backend:future().

The transaction's versionstamp, as a future that resolves after it commits.

Laziness is not an optimisation here, it is the contract. Callers capture this future before committing and wait it afterwards — set_mod_state/4 in dgen_server does exactly that to version its state cache. Resolving eagerly hands back the undefined that exists before a commit version has been assigned, and the cache then keys itself on a value that never matches, or worse, spuriously does.

is_transaction/1

-spec is_transaction(term()) -> boolean().

key_strinc(Key)

-spec key_strinc(dgen_backend:key()) -> dgen_backend:key().

on_error(Tx, Code)

-spec on_error(tx(), integer()) -> dgen_backend:future().

Resets a transaction for retry when Code is retryable, mirroring erlfdb:on_error/2: the returned future succeeds for a retryable error and raises otherwise. A reset drops staged mutations and conflict ranges, cancels the transaction's watches, and takes a fresh read version.

open/1

-spec open(atom() | #{name := atom(), seed => integer(), faults => fault_opts()}) -> db().

Opens (or resets) a named in-memory database.

Opts may carry seed for the fault RNG and faults (see set_faults/2).

reset/1

-spec reset(db()) -> ok.

Clears all data, keeping the database open. Versions keep advancing.

sandbox_open(Name, DirName)

-spec sandbox_open(term(), term()) -> dgen_backend:tenant().

Opens a database and a root directory, mirroring dgen_erlfdb:sandbox_open/2 so DGen.Case can swap backends without knowing which one it has.

set(Tx, Key, Value)

-spec set(tx(), dgen_backend:key(), binary()) -> ok.

set_faults/2

-spec set_faults(db(), fault_opts()) -> ok.

Sets the commit fault policy.

  • conflict_p — probability a commit is rejected with not_committed (1020) even though nothing actually conflicted. Retryable, so a correct caller recovers; the point is to exercise that it does.
  • commit_fail_p — probability a commit fails with a non-retryable error, which must surface to the caller rather than be retried away.

set_read_version(Tx, Version)

-spec set_read_version(tx(), integer()) -> ok.

set_versionstamped_key(Tx, Key, Value)

-spec set_versionstamped_key(tx(), dgen_backend:key(), binary()) -> ok.

set_versionstamped_value(Tx, Key, Value)

-spec set_versionstamped_value(tx(), dgen_backend:key(), binary()) -> ok.

stats/1

-spec stats(db()) -> #{atom() => non_neg_integer()}.

Commit count, conflict count, current version, and key count.

transactional/2

-spec transactional(db() | tx(), fun((tx()) -> Result)) -> Result when Result :: term().

Runs Fun in a transaction, retrying on retryable errors — the same contract as erlfdb:transactional/2. A transaction handle passed in is used directly, so nesting composes.

wait/1

-spec wait(dgen_backend:future()) -> term().

wait_for_all(Futures)

-spec wait_for_all([dgen_backend:future()]) -> [term()].

watch(Tx, Key)

-spec watch(tx(), dgen_backend:key()) -> dgen_backend:future().

Watches Key, firing {Ref, ready} at the watcher when its value changes.

A watch is anchored to the transaction that creates it, not to the moment it is registered. FoundationDB reports a change relative to the value the creating transaction would read, and getting that wrong is not a subtle infidelity — it is a silently lost wakeup:

  • A write that lands after the transaction's read version but before the watch is registered still fires it. consume_queued/5 in dgen_server reads an empty queue and then watches the push key; without this, a push committed in that window leaves the consumer asleep on a queue that already has work, and the caller's call/4 in dgen times out with nothing wrong anywhere else. The check and the registration run under the commit lock so no write can slip between them either.

  • The creating transaction's own writes do not fire it — they are part of the value it would read. push_call/7 in dgen writes the reply sentinel and watches it in one transaction; firing there costs a read-and-rewatch round trip on every call, and that round trip is precisely the window the first case then loses a reply in.

Firing happens at registration rather than at commit, which is a deliberate simplification: a watch that fires early is safe here because every consumer re-reads the watched state before acting, whereas one that fires late is not.

watch/3

-spec watch(tx(), dgen_backend:key(), list()) -> dgen_backend:future().