# Native Luau application procedures SqliteEngine 0.1 provides immutable, bounded Luau procedures for small application commands. A physical SQLite connection owns one warm Luau VM, compiled-function cache, and bounded prepared-statement cache. VMs are never shared between connections. > Luau is required native build input, not stock Lua. Run > `bin/setup_native_luau.sh` once before a source build. The pinned ref and > commit are in `native/locks/luau.lock`. ## Quick start ```elixir alias SqliteEngine.Procedures {:ok, conn} = SqliteEngine.start_link(database: "commands.db") source = ~S""" local command, context = ... local previous = db.scalar( "SELECT value FROM counters WHERE id = ?", command.counter_id ) if previous == nil then error("counter not found") end db.exec( "UPDATE counters SET value = ? WHERE id = ?", previous + command.increment, command.counter_id ) return { command_id = context.command_id, previous = previous, current = previous + command.increment, } """ {:ok, procedure} = Procedures.install(conn, source, metadata: %{"name" => "increment-counter"}) command = %{ command_id: "command-001", procedure: procedure.key, arguments: %{"counter_id" => 10, "increment" => 2}, application_time: 1_723_500_000 } {:ok, [receipt]} = Procedures.apply_batch(conn, [command], transaction: :immediate, results: :stored, security: :application_writes, deterministic: true ) {:ok, result} = Procedures.fetch_result(conn, receipt.result_ref) ``` Create application tables before applying the example. Framework tables use the private `__sqlite_engine_` prefix and are initialized transactionally and idempotently. Do not depend on their columns; public APIs are the compatibility contract. ## Identity and catalog A procedure key is exactly `SHA-256(source bytes)`. Source is authoritative and immutable. The catalog stores source, size, pinned Luau identity, runtime ABI, compile options, bounded metadata, and an optional caller-supplied installation time. Bytecode is not persisted. On a connection cache miss SqliteEngine reads one catalog row, verifies all metadata and the digest, compiles it once, and caches the loaded function. A hot call does not read source or compile. Use `Procedures.warm/3` to warm selected keys on a specific physical connection; SqliteEngine never scans and compiles the whole catalog. `info/3`, bounded `list/2`, `exists?/2`, and `delete/3` manage catalog entries. Deleting a key does not unload it from an already-warm connection. Deletion is rejected while durable receipts reference the key. ## Canonical values Arguments, contexts, SQLite cells, immediate results, durable payloads, and command hashes use canonical codec v1: | Elixir | Luau / SQLite meaning | | --- | --- | | `nil` | null / nil | | boolean | boolean | | signed 64-bit integer | exact i64; common arithmetic-range integers act as Luau numbers | | finite float | Luau number / SQLite float | | valid UTF-8 binary | text / string | | `{:blob, binary}` | BLOB / Luau `buffer` | | list | dense table | | map with UTF-8 binary keys | string-key table | Maps encode in unsigned UTF-8 key-byte order. The codec rejects unsupported terms, invalid UTF-8 text, non-finite floats, integer overflow, improper lists, non-string map keys, excessive depth/count/bytes, malformed input, sparse or mixed result tables, cycles, and multiple return values. Empty input containers carry an internal kind marker so empty maps and lists round-trip. Small exact integers use ordinary Luau arithmetic. Values outside IEEE-754's exact integer range remain Luau i64 values and round-trip exactly; use Luau's `int` library for arithmetic on those large i64 values. Integral Luau numeric results in the exact range canonicalize as integers; use a non-integral value when float identity is semantically required. Each invocation receives fresh command data. Context tables are recursively read-only and contain `command_id`, `procedure_key`, `batch_id`, and zero-based `index`, in addition to caller context. IDs, timestamps, random outcomes, and other reducer inputs must be supplied by the application. ## Database capability Procedures have one immutable `db` table: ```lua db.exec(sql, ...) db.scalar(sql, ...) db.one(sql, ...) db.all(sql, ...) db.changes() db.last_insert_rowid() ``` - `exec` accepts exactly one non-row-producing statement, exact parameters, and returns `sqlite3_changes64()`. - `scalar` requires one column and zero or one row. No row is nil; a second row is an error. - `one` returns nil or one dense indexed row and rejects a second row. - `all` returns dense indexed rows and errors rather than truncating at a limit. - Binds and columns preserve NULL, i64, finite float, UTF-8 text, and BLOB. - Containers are not SQL bind values. SQL is cached by exact bytes and security profile. The per-connection LRU holds 128 entries, resets and clears every statement, and finalizes on eviction or connection close. Use fixed parameterized SQL. Query ordering is SQLite's; include `ORDER BY` when required. ## Batches and transactions `apply_batch/3` validates every command and preflights every unique cold key before `BEGIN`. It checks out one physical connection, begins once, performs one native heterogeneous dispatch, stores receipts/results, and commits once. Commands execute in input order. The first error stops execution; conversion, execution, result, receipt, commit, or cancellation failure rolls the whole batch back. Errors use zero-based `batch_index`. There is no continue-on-error or per-command savepoint mode. Transaction modes are `:deferred`, `:immediate` (default), and `:exclusive`. Procedures cannot begin, commit, roll back, create savepoints, attach databases, load extensions, mutate schema, or access private framework tables. `apply_batch_in_transaction/3` requires an already-active checked-out connection. It neither commits nor rolls back. Warm keys before entering the transaction, or explicitly pass `allow_cold_load: true` and accept the longer lock duration. ## Idempotency and durable results A nonempty UTF-8 command ID is at most 255 bytes. Its content hash includes the procedure key, canonical arguments, caller context, result mode, security profile, and deterministic option. A committed identical ID/hash replays its receipt without procedure execution. Reusing an ID with different content is a structured `:command_id_conflict`. Duplicate IDs within a batch follow the same rule. Result modes: - `:none`: validate and discard the result; store a minimal receipt. - `:inline`: return and retain up to 64 KiB for exact replay. - `:stored`: atomically store up to 8 MiB and return a deterministic 32-byte `ResultRef`; aggregate stored payload is at most 16 MiB per batch. Effects, receipt, and result share the application transaction. Failed commands have no receipt. Use `lookup_receipt/3` to resolve an ambiguous commit response. Use `fetch_result/3`, receipt-safe `delete_result/3`, and bounded `prune_results/2`. Pruning requires an application cutoff and removes matching results and receipts; there is no host clock or background cleanup service. ## Limits, cancellation, and authority `%SqliteEngine.Procedures.Limits{}` defaults to: - 100 ms wall deadline and 1,000,000 Luau fuel checkpoints; - 8 MiB invocation memory growth; - 1,000 SQL statements and 10,000 rows; - 8 MiB each SQL input and row output; - 1 MiB arguments/results, depth 32, and 10,000 values; - 1,000 commands and a 16 MiB aggregate batch envelope. Options may narrow defaults or select values up to documented native hard maxima; they cannot exceed maxima. Errors distinguish instruction, timeout, cancellation, memory, statement, row, byte, result, argument, and security failures. SQLite progress/cancel handling covers SQLite work; Luau interrupt hooks cover pure loops. A failure rolls back the owning API and the next call can reuse the connection. `Procedures.stats/2` reports VM memory, execution, compile, and statement-cache counters. Security profiles are: - `:application_writes` (default): ordinary reads/DML, no transaction topology, schema, attach/detach, pragma, native extension, or private-table access; - `:read_only`: additionally denies inserts, updates, and deletes; - `:trusted`: broad ordinary SQL but still denies host transaction control, schema/topology escape, native extension loading, and hard-limit bypass. `deterministic: true` denies SQLite clock/random functions. The VM never exposes Luau OS time or math random facilities in any profile. Determinism still depends on SQLite/Luau versions, explicit ordering, collation, floating point, and any application-installed functions. ## Pools, telemetry, testing Every high-level pool operation is one `DBConnection` operation, so checkout, preflight, transaction, native execution, result storage, and commit use one physical connection. Reconnect loses only native caches; source remains in the SQLite catalog. Pool queue timeout (`:timeout`) is separate from procedure `limits.timeout_ms`. Telemetry events are: ```text [:sqlite_engine, :procedures, :operation, :start] [:sqlite_engine, :procedures, :operation, :stop] ``` Metadata includes operation/status and structured error phase/reason. It never contains source, SQL values, arguments, results, or full IDs/keys. Native work and memory counters are available through `stats/2` rather than payload-bearing telemetry. `SqliteEngine.Procedures.Testing` opens an initialized in-memory connection and installs/warms named test procedures without adding a test-framework dependency. For a multi-connection pool, use a file or shared-memory SQLite URI; ordinary `:memory:` creates one independent database per physical connection. ## Errors and low-level API Public failures are `%SqliteEngine.ProcedureError{}`. Match `phase`, `reason`, and `batch_index`; diagnostic text is not stable. Inspection redacts IDs and keys. Rollback failure is retained separately. Busy/checkout errors are only marked retryable when semantics prove it; callers should normally retry by command ID. `SqliteEngine.Sqlite3.prepare_luau/2`, parameterless `execute_luau/2`, argument-aware `execute_luau/4`, homogeneous `execute_luau_batch/3`, and native stats remain available as low-level controls. Low-level execution requires an active transaction and leaves rollback policy to its caller. ## Build, benchmark, and limitations Run: ```bash bin/setup_native_luau.sh bin/qa_check.sh bin/benchmark_luau_roadmap.sh epic007 bin/compare_luau_roadmap.sh BASELINE.csv CANDIDATE.csv comparison.md ``` ROADMAP001 artifacts are under `benchmark/results/roadmap001/`. Raw samples are immutable and record work dimensions; timing is evidence, never a correctness test. This feature is for bounded application procedures, not arbitrary hostile-code certification or OS process isolation. Deferred features include persistent bytecode, mutable aliases, packages/modules, background warming, queues, workflows, event outboxes, streaming cursors, custom codecs, signing/ACLs, per-command savepoints, and cross-database transactions. Precompiled support is limited to current OTP 29 / NIF 2.18 macOS and Linux x86_64/aarch64 artifacts for glibc and musl (Alpine). Windows, Android, and broad historical runtime matrices are not supported.