Capture and restore for notebook.db — the node's one durable obligation.
Backup is a notebook operation, not a subsystem beside the notebook. The
node holds exactly one database worth retaining, so "take a backup" and "back up
the notebook" name the same act, and a separate context would be a registry with
one row in it. A backup is one file: notebook-<id>.db, where <id> is
YYYY-MM-DD-HH-MM-SS in UTC.
A separate, privileged path — NOT the agent's SQL surface
YmerNode.Notebook is the agent's open-SQL surface; it deliberately blocks
ATTACH/DETACH and rolls back writes to contain prompt injection. Capture and
restore run VACUUM INTO and replace the store whole, so they live here, on
their own path, never reachable through YmerNode.Notebook.execute/1 or
query/1.
Backups are VACUUM INTO, not file copies
The database runs in WAL mode, so recent commits sit in notebook.db-wal until
checkpointed — a raw file copy would be torn or stale. VACUUM INTO writes a
transactionally consistent, compacted single file with no -wal/-shm, and it
copies the vec0 virtual tables faithfully.
What the caller may see, and what stays opaque
Ids are opaque handles. resolve/1 validates one against a strict pattern and
confirms the resolved file sits inside the backups directory before touching the
filesystem, which closes path traversal on an id arriving from a worker. Every
function reachable through the MCP tool catches File/Exqlite errors, logs the
path-bearing detail server-side, and returns a path-free atom — a caller that
forces a failure never reads back a filesystem path.
Restore is destructive, and its undo is automatic
restore/1 overwrites the entire current database. It captures an ordinary
backup of the current state first — the safety backup, whose id is returned so
the roll-back can itself be rolled back — then quiesces the pool by stopping the
repo, and swaps the file.
Order inside the down window is load-bearing, and the first step is a refusal.
The temp path beside the store is cleared of anything an earlier restore left
there — a copy, or the -wal/-shm a switch that never closed cleanly leaves
beside one — because a stale WAL beside the next copy would be replayed into
it by the very open that checks it. Then the copy is checked before
anything is overwritten: a raw handle opens notebook.db.restore-tmp and
switches it to the journal mode the pool is configured for, and a file SQLite
cannot read as a database refuses at that statement, so the restore stops with
the live store and its -wal untouched and the caller is told to choose another
backup. The refused copy is removed on the way out, so a rejected backup leaves
nothing beside the store either. Then clear the stale -wal/-shm before
renaming the new file into place, never after. Clearing them is non-negotiable
— leave a stale WAL beside the new file and SQLite replays the old WAL onto it
and corrupts it — and doing the rm first closes the crash-mid-swap window: a
kill between rename and a later rm would pair the new database with the
old WAL, the exact corruption this feature exists to prevent. With
clear-then-check-then-rm-then-rename, every crash point leaves a consistent
file — the old database with its own WAL, or the new one with no WAL — and
nothing recoverable is lost, because the safety backup already captured the
current committed state with the repo up, so its VACUUM INTO read the live
WAL.
The switch earns its place twice. A VACUUM INTO member is a rollback-journal
database, so switching the copy before the rename means the restarted pool's
first connections meet a file already in the journal mode they are configured
to set instead of racing each other to convert it — the "database is locked"
line every restore used to log. And it is what keeps a file that will never
open out of the live store at all: without it the restart answers {:ok, …}
for a repo whose every query then fails, and the standing instruction to
restart the app becomes a boot that never completes.
The safety capture is an ordinary backup and counts against retention like any other, with one exception: it is taken with the backup being restored exempt from that prune. At the retention limit the capture would otherwise drop the oldest backup, which may be the very one whose path was just resolved.
sequenceDiagram
autonumber
participant C as MCP caller (holds Lock)
participant R as YmerNode.Notebook.Repo
participant S as YmerNode.Supervisor
participant F as Filesystem
C->>F: resolve the chosen backup (before anything destructive)
C->>R: VACUUM INTO (safety backup, repo UP)
C->>S: terminate_child(repo)
C->>F: clear notebook.db.restore-tmp and its -wal/-shm (a leftover, if any)
C->>F: cp chosen backup → notebook.db.restore-tmp
C->>F: open the copy, switch it to the configured journal mode — refuses a file that is not a database
C->>F: rm notebook.db-wal, notebook.db-shm (stale, before the rename)
C->>F: rename notebook.db.restore-tmp → notebook.db (atomic, same fs)
C->>S: restart_child(repo), whatever the swap answered
Note over R: init/2 reloads vec0 on the restarted pool
C->>R: SELECT 1 — serving, not serving, or down
Note over C: the swap's answer and the restart's answer are combined lastRestart is best-effort, not an absolute guarantee. with_repo_down/2
restarts the repo whatever the body answered or raised. An untrappable kill of
the caller mid-window, or a failed restart_child, can leave the repo down
until the app restarts — the supervisor is :one_for_one and will not
auto-recreate a deliberately-terminated child. The recovery for a down repo is
an app restart, which boots onto the atomically-swapped file. The safety backup
is the data undo and presupposes a live repo; it is not the recovery path
for a down repo. The down window is one filesystem swap and the dominant
failure — the swap raising — is handled, so this residual is accepted rather
than redesigned around a supervised swap-owner.
A store the node cannot open is the operator's to recover, deliberately and
by hand; the README's layout section is the guide. A restore will not do it: the
safety capture reads the live database, so on a store that will not open there
is nothing to capture, and overwriting it with no copy would be data loss under
the node's own hand on the one store the node exists to keep. create/1 and
restore/1 therefore answer :notebook_not_serving and touch nothing.
What is not accepted is reporting independent outcomes as one. A restore has
two answers — whether the swap landed, and what the notebook can do
afterwards — and collapsing them loses the more valuable one: a restore that
landed and then failed to restart was reported as a plain failure, so the caller
retried a swap that had already happened, and the id of the backup that would
undo it went with the error. with_repo_down/2 therefore captures the body's
outcome, attempts the restart, asks the pool one statement, and returns both
facts; restore_outcome/3 combines them, and every answer that could have
changed the store names that backup.
What this module coordinates
flowchart TD
B[YmerNode.Notebook.Backup]
subgraph owned["Owned"]
LK[Backup.Lock]
end
subgraph external["External"]
NR[Notebook.Repo]
SUP[YmerNode.Supervisor]
SQ[Exqlite.Sqlite3]
FS[(backups directory)]
end
B -->|"serialises capture and restore"| LK
B -->|"VACUUM INTO, and one statement after the restart"| NR
B -->|"terminate and restart during a restore"| SUP
B -->|"opens the copy off-pool to check and switch it"| SQ
B -->|"backup files, retention"| FSConcurrency: a capture or restore is rejected while either is in progress — see
YmerNode.Notebook.Backup.Lock.
Summary
Functions
The id dt allocates in dir: YYYY-MM-DD-HH-MM-SS in UTC, advanced one
second at a time while a backup file already exists at that id. Serialized by
the lock, so this almost never iterates.
Captures a backup: one id, one VACUUM INTO, then a prune to retain/0
oldest-first. Returns {:ok, entry} — the same shape list/0 rows carry —
{:error, :operation_in_progress} if a capture or restore is already running,
{:error, :notebook_down} if the notebook is not running once the repo's own
bounded wait has passed (a capture reads the live database, so a stopped repo
has nothing to read; a repo inside a supervised restart is waited for and then
read), {:error, :notebook_not_serving} if it is running and the node cannot
open the store (the same is true, one layer down), or {:error, :backup_failed}
if the file cannot be written, in which case the partial output is removed so a
corrupt backup never appears in list/0.
Absolute path to the backups directory
(config :ymer_node, YmerNode.Notebook.Backup, :directory).
Lists backups, newest first: %{id, created_at, size_bytes}. Returns
{:error, :backups_failed} if the backups directory cannot be read — a
path-opacity boundary, since the underlying File errors embed absolute paths.
Drops the oldest backups beyond retain/0. Ids in exempt are never dropped,
even when they are the oldest.
Maps an opaque backup id to its file path. Returns {:ok, path},
{:error, :invalid_backup_id} when the id is not a bare YYYY-MM-DD-HH-MM-SS
token, or {:error, :backup_not_found}. The strict pattern and the in-directory
check together make path traversal unrepresentable — the argument arrives from
the worker.
Restores notebook.db from backup id. DESTRUCTIVE: replaces the whole
database. Captures the safety backup first — an ordinary backup of the
pre-restore state, whose id is what the caller restores to undo this restore —
then stops the repo and swaps the file. Returns
{:ok, %{restored: id, safety_backup: id}}.
Combines a with_repo_down/2 outcome for the swap with the ids the restore
holds — restored, the backup asked for, and safety_backup, the one taken
just before the swap — into the answer the caller gets.
How many backups to keep (config :ymer_node, YmerNode.Notebook.Backup, :retain,
default 40). The oldest beyond the limit are pruned after each
successful capture.
Replaces the live database with the bytes of backup src, in two phases. Up to
the point of no return: clears anything an earlier restore left at the temp
path beside the store, copies src there, and checks that the copy opens as
a database by switching it to mode, one of the journal modes
YmerNode.Notebook.Repo.is_journal_mode/1 admits. Past it: removes
the stale -wal/-shm, then renames the temp into place atomically on the
same filesystem. Order is load-bearing — see the module doc.
Stops repo, closing the pool so nothing holds the file, runs fun with the
repo DOWN, then restarts it — always, whatever fun answered or raised.
Functions
The id dt allocates in dir: YYYY-MM-DD-HH-MM-SS in UTC, advanced one
second at a time while a backup file already exists at that id. Serialized by
the lock, so this almost never iterates.
Public ONLY so the collision branch is directly testable. The scan is
load-bearing beyond id hygiene: VACUUM INTO refuses an existing target, so
reusing an id would fail the capture and send create/1 down its cleanup path,
which removes the file at that id — the pre-existing backup's.
Captures a backup: one id, one VACUUM INTO, then a prune to retain/0
oldest-first. Returns {:ok, entry} — the same shape list/0 rows carry —
{:error, :operation_in_progress} if a capture or restore is already running,
{:error, :notebook_down} if the notebook is not running once the repo's own
bounded wait has passed (a capture reads the live database, so a stopped repo
has nothing to read; a repo inside a supervised restart is waited for and then
read), {:error, :notebook_not_serving} if it is running and the node cannot
open the store (the same is true, one layer down), or {:error, :backup_failed}
if the file cannot be written, in which case the partial output is removed so a
corrupt backup never appears in list/0.
{:error, :notebook_down} is also the answer when YmerNode.Notebook.Backup.Lock
stays unreachable past its own restart window — its with_lock/2 names the
two exits behind that answer. The notebook may be serving on that route — it
is the lock that is unreachable, not the repo — but no capture or restore can
be serialised without the lock, and a node restart is the one action that
brings it back, which is what the message asks for.
Options:
:prune— setfalseto keep every existing backup.:exempt— ids this call's prune must never drop. The restore passes the id being restored: at the retention limit the safety capture would otherwise prune the very backup being restored, whichresolve/1has already turned into a path, failing the swap mid-restore.
Path-opacity boundary: a File/Exqlite error is logged server-side and never
reaches the caller as a path.
Absolute path to the backups directory
(config :ymer_node, YmerNode.Notebook.Backup, :directory).
Lists backups, newest first: %{id, created_at, size_bytes}. Returns
{:error, :backups_failed} if the backups directory cannot be read — a
path-opacity boundary, since the underlying File errors embed absolute paths.
Drops the oldest backups beyond retain/0. Ids in exempt are never dropped,
even when they are the oldest.
Public ONLY so retention is directly testable; it is called by create/1 after
a successful capture — never before, so a failed capture cannot cost an existing
backup. A removal failure is tolerated: pruning must never turn a successful
capture into an error.
Maps an opaque backup id to its file path. Returns {:ok, path},
{:error, :invalid_backup_id} when the id is not a bare YYYY-MM-DD-HH-MM-SS
token, or {:error, :backup_not_found}. The strict pattern and the in-directory
check together make path traversal unrepresentable — the argument arrives from
the worker.
Restores notebook.db from backup id. DESTRUCTIVE: replaces the whole
database. Captures the safety backup first — an ordinary backup of the
pre-restore state, whose id is what the caller restores to undo this restore —
then stops the repo and swaps the file. Returns
{:ok, %{restored: id, safety_backup: id}}.
Every answer past that capture that could have changed the store names the
safety backup, because once the swap has started overwriting it is not a
nicety: the old database has lost its -wal, so a commit made shortly before
the restore survives only there.
{:error, {:restored_notebook_down, %{restored: id, safety_backup: id}}}— the swap landed and the repo did not come back.{:error, {:restored_notebook_not_serving, %{restored: id, safety_backup: id}}}— the swap landed, the repo came back, and it cannot open the store it now has.{:error, {:backup_unreadable, %{safety_backup: id}}}— the chosen backup does not open as a database and the notebook is serving as it was: nothing was overwritten, and another backup is the way forward.{:error, {:restore_incomplete, %{safety_backup: id}}}— the swap failed after it had begun overwriting, so the safety backup is the recovery.{:error, {:restore_not_started, %{safety_backup: id}}}— nothing was touched and the notebook is serving as it was, so retrying is safe: either the repo could not be stopped, or the copy or its check stopped before the live database was overwritten.
When nothing was overwritten and the notebook did not come back, the answer
is the bare {:error, :notebook_down} or {:error, :notebook_not_serving}
instead: "retry" and "choose another backup" are instructions that cannot come
true for a notebook that cannot take a call, the store is as it was, and the
safety backup is an ordinary backup in list/0 with nothing to undo.
Before that capture there is no id to name: {:error, :operation_in_progress},
{:error, :invalid_backup_id}, {:error, :backup_not_found},
{:error, :notebook_down} when the notebook is not running once the repo's
bounded wait has passed — or when YmerNode.Notebook.Backup.Lock stays
unreachable past its own restart window (its with_lock/2 names the exits
behind that answer), where the notebook may be serving but nothing can be
serialised and a node restart is the one action that brings the lock back —
{:error, :notebook_not_serving} when it is running
and the node cannot open the store — nothing is captured and nothing is
overwritten, and recovering such a store is the operator's, by hand — and
{:error, :restore_failed} from the rescue below. That rescue sits on this
function rather than inside the swap, so a File/Exqlite error is logged
server-side and never reaches the caller as a path, whichever step raised it —
the swap's own raise is caught earlier, where the safety backup's id is still in
scope.
Combines a with_repo_down/2 outcome for the swap with the ids the restore
holds — restored, the backup asked for, and safety_backup, the one taken
just before the swap — into the answer the caller gets.
A swap that did not land splits by what it left on disk, which its reason
carries. Two reasons mean nothing was overwritten — :backup_unreadable, the
copy was refused before the first removal, and :swap_not_started, the copy or
its check stopped before it — and for those the notebook's state after the
restart decides the answer: serving, and the caller is told what was wrong with
this attempt — a different backup is the way forward for one, a retry is honest
for the other; not serving or down, and the bare :notebook_not_serving /
:notebook_down answers go back instead, because "retry" and "choose another
backup" are instructions that cannot come true for a notebook that did not
come back, and the store is as it was, so there is nothing for a safety
backup's id to undo. Any other reason means the old database is no longer
whole, and the safety backup is the recovery whatever the notebook then did.
A swap that did land splits by the notebook's serving state, and the two failure
answers go to different observers: :down is a process that is gone and an app
restart brings it back onto the restored data; :not_serving is a file the
operator has to look at, because the restart landed and the pool still cannot
open the store it now has. With the pre-swap check in front of it that second
answer is the residual of residuals — a copy that opened off-pool and that the
restarted pool still cannot serve — and it is answered rather than folded into
the first precisely because the two ask for different things.
Public ONLY so every answer is directly testable. Two of the INPUTS cannot be
forced through restore/1 from a test: Supervisor.restart_child/2 answers
{:ok, pid} even for a repo pointed at a database it cannot open, because the
pool's connections fail later and asynchronously, so a restart cannot be made to
fail from outside the window; and Supervisor.terminate_child/2's only error
needs an absent child spec, which the safety capture's own liveness check has
already refused by the time the swap runs. Two of the ANSWERS are reachable
regardless: {:restore_not_started, …} is also what a copy that failed before
anything was overwritten gets, and {:backup_unreadable, …} is what any file
that is not a database gets — an unreadable source forces both from a test.
How many backups to keep (config :ymer_node, YmerNode.Notebook.Backup, :retain,
default 40). The oldest beyond the limit are pruned after each
successful capture.
Replaces the live database with the bytes of backup src, in two phases. Up to
the point of no return: clears anything an earlier restore left at the temp
path beside the store, copies src there, and checks that the copy opens as
a database by switching it to mode, one of the journal modes
YmerNode.Notebook.Repo.is_journal_mode/1 admits. Past it: removes
the stale -wal/-shm, then renames the temp into place atomically on the
same filesystem. Order is load-bearing — see the module doc.
Returns :ok; {:error, :backup_unreadable} when the copy does not open, in
which case nothing has been overwritten and the refused copy is removed again,
so the store's directory is left exactly as it was found; or
{:error, :swap_not_started} when anything else stopped the first phase — a
leftover the node could not clear, a copy that failed — which is the same
promise about the store with a different way forward. Only the second phase
RAISES: removing the WAL is mandatory, and removal fails loudly on a real
(non-:enoent) error so a restore fails rather than corrupts. The caller MUST
have stopped the repo first. The original src is preserved — the check runs
on the copy, so no member on disk is ever mutated.
Stops repo, closing the pool so nothing holds the file, runs fun with the
repo DOWN, then restarts it — always, whatever fun answered or raised.
The body's outcome and the notebook's serving state are separate facts and come back as separate facts:
{:done, value, :serving | :not_serving | :down}—funreturnedvalue, and the notebook came back serving, came back registered but unable to open its store, or did not come back at all. A failed restart is logged server-side; it never replacesvalue.{:not_started, reason}— the repo could not be stopped, sofunnever ran and nothing was touched.Supervisor.terminate_child/2's only error is:not_found, so in practice this means the child spec has gone.
The third value is three-valued because registration is not service:
Supervisor.restart_child/2 answers {:ok, pid} for a repo whose connections
will fail asynchronously, so the restart is followed by one statement through
the pool — see restart_repo/1.
A raising fun is re-raised with its original kind and stacktrace after the
restart attempt, so an unexpected failure is never quietly converted. The restore
path catches the swap's raise in its own body before it reaches here, which is
what keeps the safety backup's id on that answer.
Public ONLY so the outcome split can be unit-tested directly; it has no MCP exposure and is called by the restore path alone. Best-effort, not an absolute guarantee — the module doc states the residual and its recovery.