YmerNode.Notebook.Backup (Ymer Node v0.2.1)

Copy Markdown View Source

Capture and restore for notebook.db — the node's one durable obligation.

Backup is a notebook operation, not a subsystem beside the notebook. The node holds exactly one database worth retaining, so "take a backup" and "back up the notebook" name the same act, and a separate context would be a registry with one row in it. A backup is one file: notebook-<id>.db, where <id> is YYYY-MM-DD-HH-MM-SS in UTC.

A separate, privileged path — NOT the agent's SQL surface

YmerNode.Notebook is the agent's open-SQL surface; it deliberately blocks ATTACH/DETACH and rolls back writes to contain prompt injection. Capture and restore run VACUUM INTO and replace the store whole, so they live here, on their own path, never reachable through YmerNode.Notebook.execute/1 or query/1.

Backups are VACUUM INTO, not file copies

The database runs in WAL mode, so recent commits sit in notebook.db-wal until checkpointed — a raw file copy would be torn or stale. VACUUM INTO writes a transactionally consistent, compacted single file with no -wal/-shm, and it copies the vec0 virtual tables faithfully.

What the caller may see, and what stays opaque

Ids are opaque handles. resolve/1 validates one against a strict pattern and confirms the resolved file sits inside the backups directory before touching the filesystem, which closes path traversal on an id arriving from a worker. Every function reachable through the MCP tool catches File/Exqlite errors, logs the path-bearing detail server-side, and returns a path-free atom — a caller that forces a failure never reads back a filesystem path.

Restore is destructive, and its undo is automatic

restore/1 overwrites the entire current database. It captures an ordinary backup of the current state first — the safety backup, whose id is returned so the roll-back can itself be rolled back — then quiesces the pool by stopping the repo, and swaps the file.

Order inside the down window is load-bearing, and the first step is a refusal. The temp path beside the store is cleared of anything an earlier restore left there — a copy, or the -wal/-shm a switch that never closed cleanly leaves beside one — because a stale WAL beside the next copy would be replayed into it by the very open that checks it. Then the copy is checked before anything is overwritten: a raw handle opens notebook.db.restore-tmp and switches it to the journal mode the pool is configured for, and a file SQLite cannot read as a database refuses at that statement, so the restore stops with the live store and its -wal untouched and the caller is told to choose another backup. The refused copy is removed on the way out, so a rejected backup leaves nothing beside the store either. Then clear the stale -wal/-shm before renaming the new file into place, never after. Clearing them is non-negotiable — leave a stale WAL beside the new file and SQLite replays the old WAL onto it and corrupts it — and doing the rm first closes the crash-mid-swap window: a kill between rename and a later rm would pair the new database with the old WAL, the exact corruption this feature exists to prevent. With clear-then-check-then-rm-then-rename, every crash point leaves a consistent file — the old database with its own WAL, or the new one with no WAL — and nothing recoverable is lost, because the safety backup already captured the current committed state with the repo up, so its VACUUM INTO read the live WAL.

The switch earns its place twice. A VACUUM INTO member is a rollback-journal database, so switching the copy before the rename means the restarted pool's first connections meet a file already in the journal mode they are configured to set instead of racing each other to convert it — the "database is locked" line every restore used to log. And it is what keeps a file that will never open out of the live store at all: without it the restart answers {:ok, …} for a repo whose every query then fails, and the standing instruction to restart the app becomes a boot that never completes.

The safety capture is an ordinary backup and counts against retention like any other, with one exception: it is taken with the backup being restored exempt from that prune. At the retention limit the capture would otherwise drop the oldest backup, which may be the very one whose path was just resolved.

sequenceDiagram
    autonumber
    participant C as MCP caller (holds Lock)
    participant R as YmerNode.Notebook.Repo
    participant S as YmerNode.Supervisor
    participant F as Filesystem
    C->>F: resolve the chosen backup (before anything destructive)
    C->>R: VACUUM INTO (safety backup, repo UP)
    C->>S: terminate_child(repo)
    C->>F: clear notebook.db.restore-tmp and its -wal/-shm (a leftover, if any)
    C->>F: cp chosen backup → notebook.db.restore-tmp
    C->>F: open the copy, switch it to the configured journal mode — refuses a file that is not a database
    C->>F: rm notebook.db-wal, notebook.db-shm (stale, before the rename)
    C->>F: rename notebook.db.restore-tmp → notebook.db (atomic, same fs)
    C->>S: restart_child(repo), whatever the swap answered
    Note over R: init/2 reloads vec0 on the restarted pool
    C->>R: SELECT 1 — serving, not serving, or down
    Note over C: the swap's answer and the restart's answer are combined last

Restart is best-effort, not an absolute guarantee. with_repo_down/2 restarts the repo whatever the body answered or raised. An untrappable kill of the caller mid-window, or a failed restart_child, can leave the repo down until the app restarts — the supervisor is :one_for_one and will not auto-recreate a deliberately-terminated child. The recovery for a down repo is an app restart, which boots onto the atomically-swapped file. The safety backup is the data undo and presupposes a live repo; it is not the recovery path for a down repo. The down window is one filesystem swap and the dominant failure — the swap raising — is handled, so this residual is accepted rather than redesigned around a supervised swap-owner.

A store the node cannot open is the operator's to recover, deliberately and by hand; the README's layout section is the guide. A restore will not do it: the safety capture reads the live database, so on a store that will not open there is nothing to capture, and overwriting it with no copy would be data loss under the node's own hand on the one store the node exists to keep. create/1 and restore/1 therefore answer :notebook_not_serving and touch nothing.

What is not accepted is reporting independent outcomes as one. A restore has two answers — whether the swap landed, and what the notebook can do afterwards — and collapsing them loses the more valuable one: a restore that landed and then failed to restart was reported as a plain failure, so the caller retried a swap that had already happened, and the id of the backup that would undo it went with the error. with_repo_down/2 therefore captures the body's outcome, attempts the restart, asks the pool one statement, and returns both facts; restore_outcome/3 combines them, and every answer that could have changed the store names that backup.

What this module coordinates

flowchart TD
    B[YmerNode.Notebook.Backup]

    subgraph owned["Owned"]
        LK[Backup.Lock]
    end

    subgraph external["External"]
        NR[Notebook.Repo]
        SUP[YmerNode.Supervisor]
        SQ[Exqlite.Sqlite3]
        FS[(backups directory)]
    end

    B -->|"serialises capture and restore"| LK
    B -->|"VACUUM INTO, and one statement after the restart"| NR
    B -->|"terminate and restart during a restore"| SUP
    B -->|"opens the copy off-pool to check and switch it"| SQ
    B -->|"backup files, retention"| FS

Concurrency: a capture or restore is rejected while either is in progress — see YmerNode.Notebook.Backup.Lock.

Summary

Functions

The id dt allocates in dir: YYYY-MM-DD-HH-MM-SS in UTC, advanced one second at a time while a backup file already exists at that id. Serialized by the lock, so this almost never iterates.

Captures a backup: one id, one VACUUM INTO, then a prune to retain/0 oldest-first. Returns {:ok, entry} — the same shape list/0 rows carry — {:error, :operation_in_progress} if a capture or restore is already running, {:error, :notebook_down} if the notebook is not running once the repo's own bounded wait has passed (a capture reads the live database, so a stopped repo has nothing to read; a repo inside a supervised restart is waited for and then read), {:error, :notebook_not_serving} if it is running and the node cannot open the store (the same is true, one layer down), or {:error, :backup_failed} if the file cannot be written, in which case the partial output is removed so a corrupt backup never appears in list/0.

Absolute path to the backups directory (config :ymer_node, YmerNode.Notebook.Backup, :directory).

Lists backups, newest first: %{id, created_at, size_bytes}. Returns {:error, :backups_failed} if the backups directory cannot be read — a path-opacity boundary, since the underlying File errors embed absolute paths.

Drops the oldest backups beyond retain/0. Ids in exempt are never dropped, even when they are the oldest.

Maps an opaque backup id to its file path. Returns {:ok, path}, {:error, :invalid_backup_id} when the id is not a bare YYYY-MM-DD-HH-MM-SS token, or {:error, :backup_not_found}. The strict pattern and the in-directory check together make path traversal unrepresentable — the argument arrives from the worker.

Restores notebook.db from backup id. DESTRUCTIVE: replaces the whole database. Captures the safety backup first — an ordinary backup of the pre-restore state, whose id is what the caller restores to undo this restore — then stops the repo and swaps the file. Returns {:ok, %{restored: id, safety_backup: id}}.

Combines a with_repo_down/2 outcome for the swap with the ids the restore holds — restored, the backup asked for, and safety_backup, the one taken just before the swap — into the answer the caller gets.

How many backups to keep (config :ymer_node, YmerNode.Notebook.Backup, :retain, default 40). The oldest beyond the limit are pruned after each successful capture.

Replaces the live database with the bytes of backup src, in two phases. Up to the point of no return: clears anything an earlier restore left at the temp path beside the store, copies src there, and checks that the copy opens as a database by switching it to mode, one of the journal modes YmerNode.Notebook.Repo.is_journal_mode/1 admits. Past it: removes the stale -wal/-shm, then renames the temp into place atomically on the same filesystem. Order is load-bearing — see the module doc.

Stops repo, closing the pool so nothing holds the file, runs fun with the repo DOWN, then restarts it — always, whatever fun answered or raised.

Functions

allocate(dt, dir)

The id dt allocates in dir: YYYY-MM-DD-HH-MM-SS in UTC, advanced one second at a time while a backup file already exists at that id. Serialized by the lock, so this almost never iterates.

Public ONLY so the collision branch is directly testable. The scan is load-bearing beyond id hygiene: VACUUM INTO refuses an existing target, so reusing an id would fail the capture and send create/1 down its cleanup path, which removes the file at that id — the pre-existing backup's.

create(opts \\ [])

Captures a backup: one id, one VACUUM INTO, then a prune to retain/0 oldest-first. Returns {:ok, entry} — the same shape list/0 rows carry — {:error, :operation_in_progress} if a capture or restore is already running, {:error, :notebook_down} if the notebook is not running once the repo's own bounded wait has passed (a capture reads the live database, so a stopped repo has nothing to read; a repo inside a supervised restart is waited for and then read), {:error, :notebook_not_serving} if it is running and the node cannot open the store (the same is true, one layer down), or {:error, :backup_failed} if the file cannot be written, in which case the partial output is removed so a corrupt backup never appears in list/0.

{:error, :notebook_down} is also the answer when YmerNode.Notebook.Backup.Lock stays unreachable past its own restart window — its with_lock/2 names the two exits behind that answer. The notebook may be serving on that route — it is the lock that is unreachable, not the repo — but no capture or restore can be serialised without the lock, and a node restart is the one action that brings it back, which is what the message asks for.

Options:

  • :prune — set false to keep every existing backup.
  • :exempt — ids this call's prune must never drop. The restore passes the id being restored: at the retention limit the safety capture would otherwise prune the very backup being restored, which resolve/1 has already turned into a path, failing the swap mid-restore.

Path-opacity boundary: a File/Exqlite error is logged server-side and never reaches the caller as a path.

directory()

Absolute path to the backups directory (config :ymer_node, YmerNode.Notebook.Backup, :directory).

list()

Lists backups, newest first: %{id, created_at, size_bytes}. Returns {:error, :backups_failed} if the backups directory cannot be read — a path-opacity boundary, since the underlying File errors embed absolute paths.

prune_to_retain(exempt \\ [])

Drops the oldest backups beyond retain/0. Ids in exempt are never dropped, even when they are the oldest.

Public ONLY so retention is directly testable; it is called by create/1 after a successful capture — never before, so a failed capture cannot cost an existing backup. A removal failure is tolerated: pruning must never turn a successful capture into an error.

resolve(id)

Maps an opaque backup id to its file path. Returns {:ok, path}, {:error, :invalid_backup_id} when the id is not a bare YYYY-MM-DD-HH-MM-SS token, or {:error, :backup_not_found}. The strict pattern and the in-directory check together make path traversal unrepresentable — the argument arrives from the worker.

restore(id)

Restores notebook.db from backup id. DESTRUCTIVE: replaces the whole database. Captures the safety backup first — an ordinary backup of the pre-restore state, whose id is what the caller restores to undo this restore — then stops the repo and swaps the file. Returns {:ok, %{restored: id, safety_backup: id}}.

Every answer past that capture that could have changed the store names the safety backup, because once the swap has started overwriting it is not a nicety: the old database has lost its -wal, so a commit made shortly before the restore survives only there.

  • {:error, {:restored_notebook_down, %{restored: id, safety_backup: id}}} — the swap landed and the repo did not come back.
  • {:error, {:restored_notebook_not_serving, %{restored: id, safety_backup: id}}} — the swap landed, the repo came back, and it cannot open the store it now has.
  • {:error, {:backup_unreadable, %{safety_backup: id}}} — the chosen backup does not open as a database and the notebook is serving as it was: nothing was overwritten, and another backup is the way forward.
  • {:error, {:restore_incomplete, %{safety_backup: id}}} — the swap failed after it had begun overwriting, so the safety backup is the recovery.
  • {:error, {:restore_not_started, %{safety_backup: id}}} — nothing was touched and the notebook is serving as it was, so retrying is safe: either the repo could not be stopped, or the copy or its check stopped before the live database was overwritten.

When nothing was overwritten and the notebook did not come back, the answer is the bare {:error, :notebook_down} or {:error, :notebook_not_serving} instead: "retry" and "choose another backup" are instructions that cannot come true for a notebook that cannot take a call, the store is as it was, and the safety backup is an ordinary backup in list/0 with nothing to undo.

Before that capture there is no id to name: {:error, :operation_in_progress}, {:error, :invalid_backup_id}, {:error, :backup_not_found}, {:error, :notebook_down} when the notebook is not running once the repo's bounded wait has passed — or when YmerNode.Notebook.Backup.Lock stays unreachable past its own restart window (its with_lock/2 names the exits behind that answer), where the notebook may be serving but nothing can be serialised and a node restart is the one action that brings the lock back — {:error, :notebook_not_serving} when it is running and the node cannot open the store — nothing is captured and nothing is overwritten, and recovering such a store is the operator's, by hand — and {:error, :restore_failed} from the rescue below. That rescue sits on this function rather than inside the swap, so a File/Exqlite error is logged server-side and never reaches the caller as a path, whichever step raised it — the swap's own raise is caught earlier, where the safety backup's id is still in scope.

restore_outcome(outcome, restored, safety_backup)

Combines a with_repo_down/2 outcome for the swap with the ids the restore holds — restored, the backup asked for, and safety_backup, the one taken just before the swap — into the answer the caller gets.

A swap that did not land splits by what it left on disk, which its reason carries. Two reasons mean nothing was overwritten — :backup_unreadable, the copy was refused before the first removal, and :swap_not_started, the copy or its check stopped before it — and for those the notebook's state after the restart decides the answer: serving, and the caller is told what was wrong with this attempt — a different backup is the way forward for one, a retry is honest for the other; not serving or down, and the bare :notebook_not_serving / :notebook_down answers go back instead, because "retry" and "choose another backup" are instructions that cannot come true for a notebook that did not come back, and the store is as it was, so there is nothing for a safety backup's id to undo. Any other reason means the old database is no longer whole, and the safety backup is the recovery whatever the notebook then did.

A swap that did land splits by the notebook's serving state, and the two failure answers go to different observers: :down is a process that is gone and an app restart brings it back onto the restored data; :not_serving is a file the operator has to look at, because the restart landed and the pool still cannot open the store it now has. With the pre-swap check in front of it that second answer is the residual of residuals — a copy that opened off-pool and that the restarted pool still cannot serve — and it is answered rather than folded into the first precisely because the two ask for different things.

Public ONLY so every answer is directly testable. Two of the INPUTS cannot be forced through restore/1 from a test: Supervisor.restart_child/2 answers {:ok, pid} even for a repo pointed at a database it cannot open, because the pool's connections fail later and asynchronously, so a restart cannot be made to fail from outside the window; and Supervisor.terminate_child/2's only error needs an absent child spec, which the safety capture's own liveness check has already refused by the time the swap runs. Two of the ANSWERS are reachable regardless: {:restore_not_started, …} is also what a copy that failed before anything was overwritten gets, and {:backup_unreadable, …} is what any file that is not a database gets — an unreadable source forces both from a test.

retain()

How many backups to keep (config :ymer_node, YmerNode.Notebook.Backup, :retain, default 40). The oldest beyond the limit are pruned after each successful capture.

swap_into_place(src, db, mode)

Replaces the live database with the bytes of backup src, in two phases. Up to the point of no return: clears anything an earlier restore left at the temp path beside the store, copies src there, and checks that the copy opens as a database by switching it to mode, one of the journal modes YmerNode.Notebook.Repo.is_journal_mode/1 admits. Past it: removes the stale -wal/-shm, then renames the temp into place atomically on the same filesystem. Order is load-bearing — see the module doc.

Returns :ok; {:error, :backup_unreadable} when the copy does not open, in which case nothing has been overwritten and the refused copy is removed again, so the store's directory is left exactly as it was found; or {:error, :swap_not_started} when anything else stopped the first phase — a leftover the node could not clear, a copy that failed — which is the same promise about the store with a different way forward. Only the second phase RAISES: removing the WAL is mandatory, and removal fails loudly on a real (non-:enoent) error so a restore fails rather than corrupts. The caller MUST have stopped the repo first. The original src is preserved — the check runs on the copy, so no member on disk is ever mutated.

with_repo_down(repo, fun)

Stops repo, closing the pool so nothing holds the file, runs fun with the repo DOWN, then restarts it — always, whatever fun answered or raised.

The body's outcome and the notebook's serving state are separate facts and come back as separate facts:

  • {:done, value, :serving | :not_serving | :down}fun returned value, and the notebook came back serving, came back registered but unable to open its store, or did not come back at all. A failed restart is logged server-side; it never replaces value.

  • {:not_started, reason} — the repo could not be stopped, so fun never ran and nothing was touched. Supervisor.terminate_child/2's only error is :not_found, so in practice this means the child spec has gone.

The third value is three-valued because registration is not service: Supervisor.restart_child/2 answers {:ok, pid} for a repo whose connections will fail asynchronously, so the restart is followed by one statement through the pool — see restart_repo/1.

A raising fun is re-raised with its original kind and stacktrace after the restart attempt, so an unexpected failure is never quietly converted. The restore path catches the swap's raise in its own body before it reaches here, which is what keeps the safety backup's id on that answer.

Public ONLY so the outcome split can be unit-tested directly; it has no MCP exposure and is called by the restore path alone. Best-effort, not an absolute guarantee — the module doc states the residual and its recovery.