ExSQL.Log (exsql v0.1.5)

Copy Markdown

A logical redo log with an async writer, the BEAM-native answer to SQLite's WAL + checkpoint.

Each committed transaction's effects are appended to a per-database log file; on open the base SQLite file is read and the log replayed on top; periodically the log is folded back into a fresh base file (a checkpoint). Write cost per commit is O(change), not O(total DB size) — unlike the whole-file rewrite of :file mode.

One writer process per base path (registered in ExSQL.LogRegistry, started under ExSQL.LogSupervisor) owns the log file. A connection casts redo records to it and returns immediately; the writer batches appends and fsyncs on a timer / on flush/1 / checkpoint / shutdown. With pool_size: 1 there is a single ordered caster, so the log is in commit order.

Record format

One record per committed transaction: term_to_binary([{sql, params}, …]), framed as <<size::32, crc32::32, payload::binary>>. A torn or crc-mismatched tail stops replay, so an interrupted append is dropped whole.

Crash-safe checkpoint

Folding the log into the base must survive a crash at any step. Done in the serialized writer (no appends interleave):

  1. fsync the active log
  2. db = read(base) + replay(log)
  3. write base.new (temp); fsync
  4. rename log → log.archived; open a fresh empty log
  5. commit point: rename base.new → base
  6. delete log.archived

Recovery decides from which files exist (see recover/1): base.new present ⇒ checkpoint didn't commit (discard it); base.new absent + log.archived present ⇒ it committed (base already folded — drop the archive). Never double-applies.

Determinism (v1, statement log)

Replay re-runs the logged SQL, so SQL-level volatile functions (random(), datetime('now')) replay differently. rowid/AUTOINCREMENT is deterministic under ordered replay. The Ecto path passes resolved values as params, so this does not affect it.

Summary

Functions

append(base_path, records)

Appends one committed transaction's statements (a list of {sql, params}).

checkpoint(base_path)

Folds the log into a fresh base file and truncates the log.

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

flush(base_path)

Forces pending appends to disk (fsync). Returns after the write is durable.

open(base_path, opts \\ [])

Ensures the writer for base_path is running and returns the recovered database.

stop(base_path)

Stops the writer for base_path (flushing first).

Functions

append(base_path, records)

@spec append(Path.t(), [{String.t(), list()}]) :: :ok

Appends one committed transaction's statements (a list of {sql, params}).

checkpoint(base_path)

@spec checkpoint(Path.t()) :: :ok

Folds the log into a fresh base file and truncates the log.

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

See Supervisor.

flush(base_path)

@spec flush(Path.t()) :: :ok

Forces pending appends to disk (fsync). Returns after the write is durable.

open(base_path, opts \\ [])

@spec open(
  Path.t(),
  keyword()
) :: {:ok, ExSQL.Database.t()} | {:error, term()}

Ensures the writer for base_path is running and returns the recovered database.

stop(base_path)

@spec stop(Path.t()) :: :ok

Stops the writer for base_path (flushing first).