YmerNode.Notebook (Ymer Node v0.2.1)

Copy Markdown View Source

The notebook — notebook.db, a separate, portable SQLite store of user- and LLM-governed memories, working and long-term: agent-defined schema, open SQL, embeddings, backup and restore. Potentially precious (tutoring records, curated knowledge); the node's obligation is durability — governance stays with the user and the LLM. This module is its raw-SQL surface (via YmerNode.Notebook.Repo).

Thin by design: the agent writes SQL; this module runs it. The only behaviour beyond pass-through is the safety model and a _meta description layer.

Down, and serving

Two different facts answer "can the notebook take my statement now", and this module reports both rather than collapsing them. Down is a fact of the process: the repo does not resolve — its name is not registered, or it is registered and Ecto has not finished starting it, a window of microseconds — so nothing can reach the store at all. Serving is a fact of the store: a statement runs now.

A notebook serves when a statement runs now. The node answers serving from what the pool itself said — a failure it holds, or one query after a restart — never from process registration, which is the other fact, down. The two come apart: a repo restarted onto a store it cannot open is registered and not serving, and reporting it as either up or down would be a lie the caller acts on.

Every action reaches the repo through ensure_meta!/0, so every action runs inside with_notebook/2. Its pre-check is YmerNode.Notebook.Repo.running?/0 — a microsecond structural read, the name and Ecto's own record of the repo behind it; its catch re-reads that, because the repo can stop between the check and the query. Serving is never probed ahead of an ordinary call. It is read only where a failure has already put the evidence in hand, through YmerNode.Notebook.Repo.not_serving?/1, which is why the happy path costs nothing.

The classification, whole, at the pre-check and again in the catch:

  • the repo resolves and the call succeeded — served, and the value goes back untouched.
  • resolves, and the pool could not provide a connection — {:error, :notebook_not_serving}. The node cannot open the store; the causes left are permissions, disk faults and an edit under a running node, and a wait fixes none of them, so the answer carries no retry advice. The node does not wait either: the pool's own reconnect backoff runs from one second toward thirty, longer than an honest answer.
  • does not resolve, with the lock held for a restore{:error, :operation_in_progress}, at once. Only a restore stops the repo, so that window is expected and lasts the file swap; it is never waited on.
  • does not resolve, anything else — a bounded wait, because a supervised restart is microseconds and answering inside one would be a terminal instruction for a condition already resolved. A capture never stops the repo, so a :backup hold beside a stopped repo is a crash rather than a window, and an unreachable lock says nothing about the repo; both wait. If the repo comes back: at the pre-check nothing has run, so the body simply runs; in the catch a :read body runs again, and a :write body does not — {:error, :notebook_restarted}, whose instruction is to check whether the statement landed, since it may already have committed and this store cannot be rebuilt.
  • still does not resolve when the wait ends — {:error, :notebook_down}, the stuck state, whose message asks for the one thing that helps, a node restart.
  • resolves, and a fault that is none of the above — back to the caller with its original kind and stacktrace, so a genuine fault is never swallowed.

One residual rides that last line, named rather than closed. The repo is resolved after the failure rather than while it happens, so a repo that stopped and came back inside a single call — a supervised restart is microseconds — resolves again by the time the catch reads it, and the failure the stopped repo caused is classified as a fault of the caller's own statement and re-raised. What the repo raises when it does not resolve is a RuntimeError, or — inside the start window — an ArgumentError out of the registry; each carries a message and nothing structural to match, so recognising either would mean reading a dependency's text — the one thing this classification does not do. The cost is one raw exception in front of a caller whose call happened to race a restart, and it closes itself: the next call finds the notebook serving.

The classification binds to structural state throughout: process registration and Ecto's own record of the repo, the lock's held op, and — for serving — an exception's documented type and reason field. Matching the text of a dependency's message stays forbidden; that text is free to change, and a check that read it would keep passing until the day the wording moved.

Read-only enforcement (non-obvious — do not remove)

query/1 runs the statement inside YmerNode.Notebook.Repo.transaction/1 and ALWAYS rolls back. A SELECT's rows ride out through the rollback value; any write a crafted statement performs is discarded. This is the hard guarantee — the leading-keyword check is only a friendly router, and is leaky on its own because WITH … DELETE is a valid write that starts with WITH. EXPLAIN reads also route here; PRAGMA is rejected (some pragmas are connection-level writes a rollback won't undo — use the tables/schema actions for introspection).

execute/1 blocks ATTACH/DETACH. The store is the only database worth protecting, and the node database sits beside it — but ATTACH reaches any SQLite file the process can open, not merely those two, so a prompt-injected agent must not have the verb at all. VACUUM INTO is blocked on the same ground — it writes a full copy of the database to any path the process can write (bare VACUUM compaction stays legal). The execute path has NO rollback backstop, so its guard is load-bearing on its own. The classifier strips ALL SQL comments and any leading whitespace/(/; and case-folds, so ;ATTACH …, (ATTACH …), and an inline attach/**/database … cannot slip past. Multi-statement tails are inert because YmerNode.Notebook.Repo.query/1 prepares only the first statement (exqlite Sqlite3.prepare) — this MUST NOT be swapped for Exqlite.Sqlite3.execute/2, which would run the tail.

Result shaping (exqlite command heuristic — do not trust it)

ecto_sqlite3 builds the query without a :command, so exqlite classifies a statement by a case-sensitive substring match for INSERT/UPDATE/DELETE. That drops the columns of a SELECT whose text contains such a substring, and reports a wrong/stale changes() count for lowercase DML and substring-bearing DDL. So query/1 passes command: :select to keep columns, and execute/1 derives affected_rows from SELECT changes() on the same connection (YmerNode.Notebook.Repo.checkout/2) keyed on our own leading-keyword classifier.

The vector layer

notebook.db loads sqlite-vec, so the agent can keep vec0 virtual tables beside ordinary ones and run KNN search through query/1. The usage discipline that comes with it — the INTEGER-PRIMARY-KEY join rule, the delete-from-both rule, and sqlite-vec's requirement that MATCH+LIMIT run in its own subquery/CTE — lives in YmerNode.Mcp.Tools.Notebook.Schemas' action notes, written for the agent that composes the SQL; help {tool: "notebook"} returns them.

_meta

SQLite has no column comments, so ensure_meta!/0 keeps a _meta(target, description, …) table (Datasette-style table:name / column:table.col keys) that tables/0 and table_schema/1 join against. It is created idempotently at the start of each call (cheap; a no-op once present) BEFORE the query rollback transaction, so a cold query cannot roll its creation away.

Summary

Functions

Runs a single DDL/DML statement (no result rows). Rejects ATTACH/DETACH, and VACUUM INTO while allowing bare VACUUM. Returns {:ok, %{affected_rows: n}} or {:error, reason}.

Runs a read-only SELECT/WITH/EXPLAIN statement inside an always-rolled-back transaction. Returns {:ok, %{columns: [...], rows: [...]}} or {:error, reason} (:not_read_only if the statement is not a read — PRAGMA is rejected; use the tables/schema introspection actions instead). Columns are recovered via command: :select regardless of exqlite's command heuristic.

Full schema for one table: %{table, virtual, description, columns, indexes}. Regular tables use PRAGMA table_info/index_list; vec0 virtual tables parse their CREATE VIRTUAL TABLE … vec0(…) DDL. Returns {:error, :table_not_found} if the table does not exist.

Lists user tables (excludes _meta, sqlite_* internals, and sqlite-vec shadow tables). Each entry: %{name, virtual, row_count, description}. Virtual (vec0) tables report row_count: nil. Shadow tables are detected by name prefix "<virtual_table>_" — keep companion tables outside that prefix (the documented pattern names them confluence_pages next to vec_confluence_pages).

Runs fun and answers a notebook that cannot take it instead of raising at it.

Functions

execute(sql)

Runs a single DDL/DML statement (no result rows). Rejects ATTACH/DETACH, and VACUUM INTO while allowing bare VACUUM. Returns {:ok, %{affected_rows: n}} or {:error, reason}.

affected_rows is the exact SELECT changes() count when the statement's leading keyword is INSERT/UPDATE/DELETE, and 0 for everything else — DDL, and the rare CTE-led WITH … DELETE write, reported conservatively as 0. It is derived from our own comment-stripped, case-folded leading-keyword classifier plus changes() on the same connection, NOT from exqlite's result, whose command classification is an unreliable case-sensitive substring match.

query(sql)

Runs a read-only SELECT/WITH/EXPLAIN statement inside an always-rolled-back transaction. Returns {:ok, %{columns: [...], rows: [...]}} or {:error, reason} (:not_read_only if the statement is not a read — PRAGMA is rejected; use the tables/schema introspection actions instead). Columns are recovered via command: :select regardless of exqlite's command heuristic.

table_schema(table)

Full schema for one table: %{table, virtual, description, columns, indexes}. Regular tables use PRAGMA table_info/index_list; vec0 virtual tables parse their CREATE VIRTUAL TABLE … vec0(…) DDL. Returns {:error, :table_not_found} if the table does not exist.

tables()

Lists user tables (excludes _meta, sqlite_* internals, and sqlite-vec shadow tables). Each entry: %{name, virtual, row_count, description}. Virtual (vec0) tables report row_count: nil. Shadow tables are detected by name prefix "<virtual_table>_" — keep companion tables outside that prefix (the documented pattern names them confluence_pages next to vec_confluence_pages).

Answers {:error, reason} while the notebook is down or cannot open its store. That is the one return here that is not a list, so match it before treating the answer as one.

with_notebook(body, fun)

Runs fun and answers a notebook that cannot take it instead of raising at it.

body is what the caller is doing, not what the guard should do with it: :read for the three actions that only read, :write for execute/1. The guard owns the consequence — only a read may be run a second time, because a write's statement may already have committed before the failure reached here, and this is the one store the node cannot rebuild.

Every action above reaches the repo, so every action runs inside this guard. The pre-check answers the ordinary stopped case without raising at all; the catch covers the race where the repo stops between the check and the query, and both classify from structural state rather than from the exception's text, which belongs to a dependency. Anything caught while the repo is serving goes back exactly as it arrived — kind, value and stacktrace — so a genuine fault is never swallowed; the moduledoc names the residual that rides that rule, a failure raised while the repo was gone and read once the repo is back. The moduledoc's "Down, and serving" section owns the whole classification and what each answer means.

Public ONLY so both halves of that catch are directly testable: the four actions hand it a fixed body, so a test working through them can neither make one raise while the repo stays up nor stop the repo from inside one. No MCP exposure.