# Implementation notes

[Back to README](../README.md#documentation).

The Rust software development kit (SDK) supplies the database engine. The
native library supplies Natively Implemented Functions (NIFs) to the Erlang
virtual machine (BEAM).

## Internal responsibilities

Module names and local comments describe the implementation. Use these
boundaries when you change the code:

| Module or component | Responsibility |
| --- | --- |
| Public Elixir modules | Check API arguments and create one operation span. |
| `Options` | Share argument checks for names, versions, and metadata. |
| `Execution` | Convert failures to `Error`. Call the operation's decoder for write results. |
| `Backend` and its codecs | Extract resource handles, encode requests, and wrap native resources. |
| `Query.Definition`, `Controls`, and `Request` | Keep semantic controls, check combinations, and create an execution plan. |
| `Query.Executor` | Use the same Arrow cursor for immediate and streaming reads. Keep callbacks in BEAM. |
| `Creation`, `Write`, `SchemaEvolution`, and `Ingestion` | Share source consumption, row and batch transport, and writer cleanup. |
| `Schema`, `ArrowSchema`, and native `StorageSchema` | Separate logical definitions, physical output, and checks for stored fields. |
| `TableCache` | Retain handles, coalesce requests, enforce capacities, and discard stale task results. |
| `TableCache.Options` and `TableCache.Worker` | Validate fixed instance settings and perform connection, open, and warming work outside the coordinator. |
| Native `outcome.rs` and `wire.rs` | Define engine results and encode them for BEAM. |
| Native compatibility modules | Keep SDK workarounds separate from the public API. |

Public builders reject unknown options. Native code checks record values
against the schema. Native query request decoding can ignore unknown fields.
Keep public option checks in the builders.

Row input uses direct BEAM terms. Native decoding creates owned values
before background work starts. Small schema and query descriptions use JSON.
`Batch.to_rows/1` converts Arrow output through JSON. Native codec tests
check JSON round trips.

A batch slice can keep its parent buffers. The `batch_size` option limits
returned rows. It does not limit memory use. Arrow IPC import and export
allocate memory. Native batch transfer avoids row conversion.

Each snapshot needs a separate SDK table and dataset view. An `Arc` or SDK
table clone shares its view. The shared Tokio runtime executes native
operations. Direct APIs create no process for each table and use no
connection pool. An optional application-supervised `TableCache` owns one
coordinator and a linked task supervisor. Its bounded tasks perform storage
work; returned handles go directly to the ordinary query APIs.

Each cache entry has a fresh generation reference. Task results apply only to
the matching generation, so invalidation and eviction cannot be undone by a
late open or warmer. Active work remains counted until it settles, even if its
entry is discarded. Caller monitors and fetch timeouts remove waiters without
cancelling a shared open. Pending fetches protect their entries from eviction.

Warming pins an independent snapshot sharing the retained handle's session.
The completion key combines the table version and the SDK's first index segment
UUID. A first-segment UUID alone misses appended index segments; the SDK's
`index_version` describes its storage format. Rechecking the live version after
warming rejects observed concurrent changes. Detection follows the configured
read-consistency interval. No completion record promises cache residency.

Connect and table open use a monitored one-shot Tokio task. Read admission
happens before spawning; one deadline covers connection/session setup and the
entire SDK operation. Reference-tagged replies preserve the synchronous Elixir
API without retaining a BEAM scheduler. Timeout, caller death, and caught task
panics drop the operation future and release its permit before replying. SDK
tasks spawned separately can outlive that future.

Blocking table/index inspection, index prewarming and deletion, and table
listing/deletion use dirty I/O schedulers. Their storage waits retain an I/O
thread while leaving dirty CPU schedulers available for Arrow conversion.
The embedded table URI accessor only copies an in-memory string. CPU-heavy
operations such as index construction and row codecs use dirty CPU schedulers.

Read cancellation and write completion have different guarantees. Explicit
cursor close waits for cleanup and read slot release. A writer abort sends
an input error and waits for SDK execution. Dropping a write future can
leave SDK tasks running separately. Cancellation cannot guarantee rollback
after final commit starts.

Backfills keep the final batch until successful source EOF. The SDK can stop
reading after it receives the expected row count. Checking EOF prevents a
late input failure from being lost. Empty backfills also check EOF before
committing.

Contract tests check snapshot isolation, schema conversion, source formats,
permit release, errors, and telemetry. Update these tests when an operation
contract changes.

## SDK compatibility

These adapters support LanceDB 0.37.1 and Lance 10. Review them when you
upgrade the SDK. Remove an adapter only after its regression tests pass
without it. Paths in this table are relative to `native/lancedb_nif/src/`,
except for Elixir builders.

| Adapter | SDK limitation and tests | Condition for removal |
| --- | --- | --- |
| `compat/storage.rs` | Null structs with required descendants are unsupported. Custom vector item properties are lost. Nested casts and escaped names can change field identity. Native schema and value tests, plus schema, evolution, and contract tests cover these cases. | The SDK keeps field properties and converts the selected types safely. |
| `compat/table.rs` | Merge and tag changes can bypass snapshot write protection. History, tag, and branch tests cover this guard. | Every SDK write operation rejects historical handles. |
| `compat/update.rs` | Row updates fail on blob tables, including updates to other columns. Storage contract tests check rejection before execution and unchanged data and version. | SDK row updates keep blob payloads and schema. |
| `compat/fts.rs` | JSON tokenization and indexing can panic on invalid input. Advanced FTS tests check invalid query triples and synchronous and asynchronous index creation. | The SDK returns typed errors for invalid input without a panic. |
| `compat/ipc.rs` | Arrow IPC schema conversion can panic on malformed metadata. IPC safety tests check missing schema fields, invalid integer widths, and panics during iteration after a valid batch. Import returns an invalid-data error for the entire stream. | Arrow returns typed errors for malformed IPC metadata without a panic. |
| `compat/statistics.rs` | Typed statistics do not include IVF training loss for individual segments. Native tests check complete, missing, and non-finite values. | Typed SDK statistics supply the same complete loss value. |
| `hybrid.rs` | Hybrid fusion can produce an incorrect schema when only text search returns candidates. Ranking and FTS tests check empty candidate sets. | Native hybrid queries keep matching projections for all empty candidate sets. |
| `projection.rs` | Ordering can need columns that projection hides. Constant expressions also need physical input. Projection and ranking tests cover these cases. | The SDK planner handles these projections directly. |
| `diagnostics.rs` | Query analysis can ignore result stream errors. Diagnostics tests check execution failures. | SDK analysis returns all result stream failures. |
| `fts.rs` and Elixir FTS builders | Multi-match serialization does not include the operator. Default column selection misses negative leaves. Fuzzy prefixes can split Unicode characters. FTS and tokenizer tests cover these cases. | SDK queries keep operators and select every leaf safely. |
| `value.rs` | Nested literals need array functions and explicit child casts. Struct literals cannot target required children. Nested update tests check exact values and unsupported errors. | Native literal updates keep types, names, nullability, and timestamps directly. |
| `cloning.rs` | The SDK loses storage overrides when opening a clone. S3 clone tests check saved connection options. | SDK cloning keeps the connection configuration. |
| `creation.rs` and `table_open.rs` | Local SDK connections discard inherited storage options for remote locations. S3 tests cover creation formats, empty tables, and overrides. | Local connections keep inherited options for all location overrides. |
| `query/vector_input.rs` | SDK builders can convert vectors to float32. Document multivector search supports only float32. Vector tests check float64 precision and Hamming input. | SDK builders keep vector types, including multivector types. |
| `ingestion/worker.rs` and bulk readers | Cancellation needs an input error and a wait for execution. Repeated input requests after EOF must still return EOF. Backfills must await source EOF. Native worker and ingestion tests cover these cases. | The SDK guarantees cancellation without separate commits and equivalent source termination. |

FTS boost subtraction, phrase positions, and explicit fuzzy distance follow
SDK behavior. Blob query results stay descriptors. Callers must fetch
payload bytes explicitly, even if an SDK upgrade changes descriptor storage.