Native Luau application procedures

SqliteEngine 0.1 provides immutable, bounded Luau procedures for small application commands. A physical SQLite connection owns one warm Luau VM, compiled-function cache, and bounded prepared-statement cache. VMs are never shared between connections.

Luau is required native build input, not stock Lua. Run bin/setup_native_luau.sh once before a source build. The pinned ref and commit are in native/locks/luau.lock.

Quick start

alias SqliteEngine.Procedures

{:ok, conn} = SqliteEngine.start_link(database: "commands.db")

source = ~S"""
local command, context = ...
local previous = db.scalar(
    "SELECT value FROM counters WHERE id = ?",
    command.counter_id
)

if previous == nil then
    error("counter not found")
end

db.exec(
    "UPDATE counters SET value = ? WHERE id = ?",
    previous + command.increment,
    command.counter_id
)

return {
    command_id = context.command_id,
    previous = previous,
    current = previous + command.increment,
}
"""

{:ok, procedure} =
  Procedures.install(conn, source, metadata: %{"name" => "increment-counter"})

command = %{
  command_id: "command-001",
  procedure: procedure.key,
  arguments: %{"counter_id" => 10, "increment" => 2},
  application_time: 1_723_500_000
}

{:ok, [receipt]} =
  Procedures.apply_batch(conn, [command],
    transaction: :immediate,
    results: :stored,
    security: :application_writes,
    deterministic: true
  )

{:ok, result} = Procedures.fetch_result(conn, receipt.result_ref)

Create application tables before applying the example. Framework tables use the private __sqlite_engine_ prefix and are initialized transactionally and idempotently. Do not depend on their columns; public APIs are the compatibility contract.

Identity and catalog

A procedure key is exactly SHA-256(source bytes). Source is authoritative and immutable. The catalog stores source, size, pinned Luau identity, runtime ABI, compile options, bounded metadata, and an optional caller-supplied installation time. Bytecode is not persisted.

On a connection cache miss SqliteEngine reads one catalog row, verifies all metadata and the digest, compiles it once, and caches the loaded function. A hot call does not read source or compile. Use Procedures.warm/3 to warm selected keys on a specific physical connection; SqliteEngine never scans and compiles the whole catalog.

info/3, bounded list/2, exists?/2, and delete/3 manage catalog entries. Deleting a key does not unload it from an already-warm connection. Deletion is rejected while durable receipts reference the key.

Canonical values

Arguments, contexts, SQLite cells, immediate results, durable payloads, and command hashes use canonical codec v1:

Maps encode in unsigned UTF-8 key-byte order. The codec rejects unsupported terms, invalid UTF-8 text, non-finite floats, integer overflow, improper lists, non-string map keys, excessive depth/count/bytes, malformed input, sparse or mixed result tables, cycles, and multiple return values. Empty input containers carry an internal kind marker so empty maps and lists round-trip.

Small exact integers use ordinary Luau arithmetic. Values outside IEEE-754's exact integer range remain Luau i64 values and round-trip exactly; use Luau's int library for arithmetic on those large i64 values. Integral Luau numeric results in the exact range canonicalize as integers; use a non-integral value when float identity is semantically required.

Each invocation receives fresh command data. Context tables are recursively read-only and contain command_id, procedure_key, batch_id, and zero-based index, in addition to caller context. IDs, timestamps, random outcomes, and other reducer inputs must be supplied by the application.

Database capability

Procedures have one immutable db table:

db.exec(sql, ...)
db.scalar(sql, ...)
db.one(sql, ...)
db.all(sql, ...)
db.changes()
db.last_insert_rowid()

• exec accepts exactly one non-row-producing statement, exact parameters, and returns sqlite3_changes64().
• scalar requires one column and zero or one row. No row is nil; a second row is an error.
• one returns nil or one dense indexed row and rejects a second row.
• all returns dense indexed rows and errors rather than truncating at a limit.
• Binds and columns preserve NULL, i64, finite float, UTF-8 text, and BLOB.
• Containers are not SQL bind values.

SQL is cached by exact bytes and security profile. The per-connection LRU holds 128 entries, resets and clears every statement, and finalizes on eviction or connection close. Use fixed parameterized SQL. Query ordering is SQLite's; include ORDER BY when required.

Batches and transactions

apply_batch/3 validates every command and preflights every unique cold key before BEGIN. It checks out one physical connection, begins once, performs one native heterogeneous dispatch, stores receipts/results, and commits once. Commands execute in input order. The first error stops execution; conversion, execution, result, receipt, commit, or cancellation failure rolls the whole batch back. Errors use zero-based batch_index. There is no continue-on-error or per-command savepoint mode.

Transaction modes are :deferred, :immediate (default), and :exclusive. Procedures cannot begin, commit, roll back, create savepoints, attach databases, load extensions, mutate schema, or access private framework tables.

apply_batch_in_transaction/3 requires an already-active checked-out connection. It neither commits nor rolls back. Warm keys before entering the transaction, or explicitly pass allow_cold_load: true and accept the longer lock duration.

Idempotency and durable results

A nonempty UTF-8 command ID is at most 255 bytes. Its content hash includes the procedure key, canonical arguments, caller context, result mode, security profile, and deterministic option. A committed identical ID/hash replays its receipt without procedure execution. Reusing an ID with different content is a structured :command_id_conflict. Duplicate IDs within a batch follow the same rule.

Result modes:

• :none: validate and discard the result; store a minimal receipt.
• :inline: return and retain up to 64 KiB for exact replay.
• :stored: atomically store up to 8 MiB and return a deterministic 32-byte ResultRef; aggregate stored payload is at most 16 MiB per batch.

Effects, receipt, and result share the application transaction. Failed commands have no receipt. Use lookup_receipt/3 to resolve an ambiguous commit response. Use fetch_result/3, receipt-safe delete_result/3, and bounded prune_results/2. Pruning requires an application cutoff and removes matching results and receipts; there is no host clock or background cleanup service.

Limits, cancellation, and authority

%SqliteEngine.Procedures.Limits{} defaults to:

• 100 ms wall deadline and 1,000,000 Luau fuel checkpoints;
• 8 MiB invocation memory growth;
• 1,000 SQL statements and 10,000 rows;
• 8 MiB each SQL input and row output;
• 1 MiB arguments/results, depth 32, and 10,000 values;
• 1,000 commands and a 16 MiB aggregate batch envelope.

Options may narrow defaults or select values up to documented native hard maxima; they cannot exceed maxima. Errors distinguish instruction, timeout, cancellation, memory, statement, row, byte, result, argument, and security failures. SQLite progress/cancel handling covers SQLite work; Luau interrupt hooks cover pure loops. A failure rolls back the owning API and the next call can reuse the connection. Procedures.stats/2 reports VM memory, execution, compile, and statement-cache counters.

Security profiles are:

• :application_writes (default): ordinary reads/DML, no transaction topology, schema, attach/detach, pragma, native extension, or private-table access;
• :read_only: additionally denies inserts, updates, and deletes;
• :trusted: broad ordinary SQL but still denies host transaction control, schema/topology escape, native extension loading, and hard-limit bypass.

deterministic: true denies SQLite clock/random functions. The VM never exposes Luau OS time or math random facilities in any profile. Determinism still depends on SQLite/Luau versions, explicit ordering, collation, floating point, and any application-installed functions.

Pools, telemetry, testing

Every high-level pool operation is one DBConnection operation, so checkout, preflight, transaction, native execution, result storage, and commit use one physical connection. Reconnect loses only native caches; source remains in the SQLite catalog. Pool queue timeout (:timeout) is separate from procedure limits.timeout_ms.

Telemetry events are:

[:sqlite_engine, :procedures, :operation, :start]
[:sqlite_engine, :procedures, :operation, :stop]

Metadata includes operation/status and structured error phase/reason. It never contains source, SQL values, arguments, results, or full IDs/keys. Native work and memory counters are available through stats/2 rather than payload-bearing telemetry.

SqliteEngine.Procedures.Testing opens an initialized in-memory connection and installs/warms named test procedures without adding a test-framework dependency. For a multi-connection pool, use a file or shared-memory SQLite URI; ordinary :memory: creates one independent database per physical connection.

Errors and low-level API

Public failures are %SqliteEngine.ProcedureError{}. Match phase, reason, and batch_index; diagnostic text is not stable. Inspection redacts IDs and keys. Rollback failure is retained separately. Busy/checkout errors are only marked retryable when semantics prove it; callers should normally retry by command ID.

SqliteEngine.Sqlite3.prepare_luau/2, parameterless execute_luau/2, argument-aware execute_luau/4, homogeneous execute_luau_batch/3, and native stats remain available as low-level controls. Low-level execution requires an active transaction and leaves rollback policy to its caller.

Build, benchmark, and limitations

Run:

bin/setup_native_luau.sh
bin/qa_check.sh
bin/benchmark_luau_roadmap.sh epic007
bin/compare_luau_roadmap.sh BASELINE.csv CANDIDATE.csv comparison.md

ROADMAP001 artifacts are under benchmark/results/roadmap001/. Raw samples are immutable and record work dimensions; timing is evidence, never a correctness test.

This feature is for bounded application procedures, not arbitrary hostile-code certification or OS process isolation. Deferred features include persistent bytecode, mutable aliases, packages/modules, background warming, queues, workflows, event outboxes, streaming cursors, custom codecs, signing/ACLs, per-command savepoints, and cross-database transactions. Precompiled support is limited to current OTP 29 / NIF 2.18 macOS and Linux x86_64/aarch64 artifacts for glibc and musl (Alpine). Windows, Android, and broad historical runtime matrices are not supported.