LanceDB.SchemaEvolution (LanceDB v0.2.0)

Copy Markdown View Source

SQL and Expr transformations, column sources and snapshot backfills.

Summary

Functions

Adds columns from SQL strings or LanceDB.Expr values for existing rows.

Adds columns from a Source with an explicit output schema in physical row order. Supply one output row per visible row, excluding deleted rows. The optional positive :expected_version guards against changes to the target version. Row counts and successful source EOF are checked before committing.

Renames columns, changes nullability, or converts top-level column types.

Maps snapshot batches to new columns, preserving row count and order.

Drops columns or nested fields and returns a WriteResult with the committed version.

Types

result(value)

@type result(value) :: LanceDB.result(value)

table()

@type table() :: LanceDB.Table.t()

Functions

add_columns(table, expressions)

@spec add_columns(table(), map() | list()) :: result(LanceDB.WriteResult.t())

Adds columns from SQL strings or LanceDB.Expr values for existing rows.

Supply a map or list of {name, expression} pairs, for example: [doubled: "score * 2", status: "'new'", note: "CAST(NULL AS STRING)"]. String expressions are SQL, including quoted string literals. Result types and field properties must meet the package's storage requirements. The operation fills existing rows once. It does not set defaults for future rows or create generated columns. Each expression uses the schema before the change. The result is a WriteResult with the new version. The operation cannot overwrite existing columns.

add_from_source(table, source, opts \\ [])

@spec add_from_source(table(), LanceDB.Source.t(), keyword()) ::
  result(LanceDB.WriteResult.t())

Adds columns from a Source with an explicit output schema in physical row order. Supply one output row per visible row, excluding deleted rows. The optional positive :expected_version guards against changes to the target version. Row counts and successful source EOF are checked before committing.

alter_columns(table, changes)

@spec alter_columns(table(), map() | list()) :: result(LanceDB.WriteResult.t())

Renames columns, changes nullability, or converts top-level column types.

Supply a map or list of {name, options} pairs, such as: [text: [rename: :title], score: [nullable: false]]. Each entry must specify :rename, :nullable, or :type. The :type option accepts a LanceDB.Schema type. A change to non-nullable fails if null values exist. Rename and nullability changes accept nested paths, such as "data.count". Use Lance backtick syntax to quote literal dots and backticks. Rename targets must be leaf names without dots. The operation rejects duplicate paths, overlapping paths, and duplicate names under the same parent before it changes the schema.

Type conversions use the SDK's compatible type families. These include integer to integer, float to float, compatible date and time types, and compatible complete lists. Drop indexes on converted columns before the conversion. Create the indexes again after the conversion. Integer overflow returns an error without a commit. Float and time conversions can lose precision. Nested field casts return an error because Lance 10 panics on these conversions. Casts must also keep the field's extension markers compatible with its storage type. The result is a WriteResult with the new version.

backfill_columns(table, schema, mapper, opts \\ [])

@spec backfill_columns(
  table(),
  LanceDB.Schema.t() | list(),
  (LanceDB.Batch.t() -> LanceDB.Batch.t() | [map()]),
  keyword()
) :: result(LanceDB.WriteResult.t())

Maps snapshot batches to new columns, preserving row count and order.

The output schema accepts a LanceDB.Schema or a list of field definitions. The mapper runs in the caller and returns a native Batch or row list for the new columns. :read_columns selects input fields; :batch_size defaults to

  1. The operation guards the snapshot version and checks successful source EOF before committing. Returns a LanceDB.WriteResult.

drop_columns(table, columns)

@spec drop_columns(table(), list()) :: result(LanceDB.WriteResult.t())

Drops columns or nested fields and returns a WriteResult with the committed version.

Indexes on dropped columns are removed by the SDK. Current rows retain their remaining fields. Old versions may retain the dropped data; compaction and version cleanup are separate operations. At least one column must remain. Paths use the same syntax as alter_columns/2; overlapping paths are rejected.