ExSQL.
Copy Markdown
A logical redo log with an async writer, the BEAM-native answer to SQLite's WAL + checkpoint.
Each committed transaction's effects are appended to a per-database log file;
on open the base SQLite file is read and the log replayed on top; periodically
the log is folded back into a fresh base file (a checkpoint). Write cost per
commit is O(change), not O(total DB size) — unlike the whole-file rewrite of
:file mode.
One writer process per base path (registered in ExSQL.LogRegistry, started
under ExSQL.LogSupervisor) owns the log file. A connection casts redo
records to it and returns immediately; the writer batches appends and fsyncs
on a timer / on flush/1 / checkpoint / shutdown.
Record format
One record per committed transaction: term_to_binary([{sql, params}, …]),
framed as <<size::32, crc32::32, payload::binary>>. A torn or
crc-mismatched tail stops replay, so an interrupted append is dropped whole.
Connections append whole transactions (the driver buffers statements between
BEGIN and COMMIT), so records from a pool of connections never interleave
mid-transaction and replay stays in commit order. Each processed append bumps
a per-path commit version in ExSQL.Registry (under version_key/1), which
lets other connections on the same path detect that their snapshot is stale
and reload via fetch/1.
Crash-safe checkpoint
Folding the log into the base must survive a crash at any step. Done in the serialized writer (no appends interleave):
- fsync the active log
db = read(base) + replay(log)- write
base.new(temp); fsync - rename
log→log.archived; open a fresh emptylog - commit point: rename
base.new→base - delete
log.archived
Recovery decides from which files exist (see recover/1): base.new present
⇒ checkpoint didn't commit (discard it); base.new absent + log.archived
present ⇒ it committed (base already folded — drop the archive). Never
double-applies.
Determinism (v1, statement log)
Replay re-runs the logged SQL, so SQL-level volatile functions (random(),
datetime('now')) replay differently. rowid/AUTOINCREMENT is deterministic
under ordered replay. The Ecto path passes resolved values as params, so this
does not affect it.
Summary
Functions
Appends one committed transaction's statements (a list of {sql, params})
and returns the bumped commit version (or :no_registry).
Folds the log into a fresh base file and truncates the log.
Returns a specification to start this module under a supervisor.
Returns the current database (base + replayed log) and its commit version.
Forces pending appends to disk (fsync). Returns after the write is durable.
Ensures the writer for base_path is running and returns the recovered database.
Stops the writer for base_path (flushing first).
The ExSQL.Registry key holding the commit version for base_path.
Functions
@spec append(Path.t(), [{String.t(), list()}]) :: non_neg_integer() | :no_registry | :ok
Appends one committed transaction's statements (a list of {sql, params})
and returns the bumped commit version (or :no_registry).
The append itself is a cast; the version is bumped here, after the cast, so
by the time another connection can observe the new version its fetch/1
call is guaranteed to queue behind this append in the writer's mailbox.
@spec checkpoint(Path.t()) :: :ok
Folds the log into a fresh base file and truncates the log.
Returns a specification to start this module under a supervisor.
See Supervisor.
@spec fetch(Path.t()) :: {ExSQL.Database.t(), non_neg_integer() | :no_registry}
Returns the current database (base + replayed log) and its commit version.
The call is serialized behind pending appends in the writer's mailbox, so the returned database reflects every append whose version bump was visible when the caller decided to fetch.
@spec flush(Path.t()) :: :ok
Forces pending appends to disk (fsync). Returns after the write is durable.
@spec open( Path.t(), keyword() ) :: {:ok, ExSQL.Database.t()} | {:error, term()}
Ensures the writer for base_path is running and returns the recovered database.
@spec stop(Path.t()) :: :ok
Stops the writer for base_path (flushing first).
The ExSQL.Registry key holding the commit version for base_path.