LanceDB.Maintenance (LanceDB v0.2.0)

Copy Markdown View Source

Table statistics, compaction, index optimization and version cleanup.

Summary

Functions

Permanently removes old versions and their unreferenced files.

Merges small data fragments and returns a Maintenance.Compaction struct. :target_rows_per_fragment defaults to 1,048,576. Positive uint32 sizes also include :max_rows_per_group, :max_bytes_per_file, :num_threads, :batch_size, :io_buffer_size, :binary_copy_read_batch_bytes, and :max_source_fragments. Boolean controls are :materialize_deletions and :defer_index_remap; :materialize_deletions_threshold is between 0 and 1. :compaction_mode is :reencode, :try_binary_copy, or :force_binary_copy; forced binary copying fails for incompatible fragments. :index_remap_mode is :direct or :compact. :max_overlays_per_fragment is nil or a non-negative uint32. Omitted options preserve SDK defaults. Current rows and schema are preserved; old files remain until version cleanup.

Migrates manifest paths to V2. Run only while all other table operations are stopped; idempotent.

Adds unindexed rows to existing indexes, returning {:ok, :optimized}. :index_names restricts work to named indexes (a non-empty list). :num_indices_to_merge controls index deltas to merge; zero only indexes new rows, nil/omitted uses the SDK default. :retrain rebuilds vector models (default false) and cannot be combined with :num_indices_to_merge. Missing indexes are not created. Use LanceDB.Index.stats/2 to inspect coverage.

Returns table statistics as a map with string keys.

Returns whether the table uses V2 manifest paths.

Types

result(value)

@type result(value) :: LanceDB.result(value)

table()

@type table() :: LanceDB.Table.t()

Functions

cleanup_old_versions(table, opts \\ [])

@spec cleanup_old_versions(table(), keyword()) :: result(map())

Permanently removes old versions and their unreferenced files.

:older_than_seconds sets retention (default 604,800: seven days). Zero is allowed for an idle table with no readers using old versions. Choose retention longer than the lifetime of readers and writers; pruned versions cannot be read again. The latest version is retained. :error_if_tagged_old_versions defaults to true; false skips tagged versions while pruning other eligible versions. :delete_unverified defaults to false. True also deletes unreferenced files within the SDK's seven-day protection window; use only with no active writers. Returns removal statistics with string keys.

compact_files(table, opts \\ [])

@spec compact_files(table(), keyword()) :: result(LanceDB.Maintenance.Compaction.t())

Merges small data fragments and returns a Maintenance.Compaction struct. :target_rows_per_fragment defaults to 1,048,576. Positive uint32 sizes also include :max_rows_per_group, :max_bytes_per_file, :num_threads, :batch_size, :io_buffer_size, :binary_copy_read_batch_bytes, and :max_source_fragments. Boolean controls are :materialize_deletions and :defer_index_remap; :materialize_deletions_threshold is between 0 and 1. :compaction_mode is :reencode, :try_binary_copy, or :force_binary_copy; forced binary copying fails for incompatible fragments. :index_remap_mode is :direct or :compact. :max_overlays_per_fragment is nil or a non-negative uint32. Omitted options preserve SDK defaults. Current rows and schema are preserved; old files remain until version cleanup.

migrate_manifest_paths_v2(table)

@spec migrate_manifest_paths_v2(table()) :: result(:migrated)

Migrates manifest paths to V2. Run only while all other table operations are stopped; idempotent.

optimize_indices(table, opts \\ [])

@spec optimize_indices(table(), keyword()) :: result(:optimized)

Adds unindexed rows to existing indexes, returning {:ok, :optimized}. :index_names restricts work to named indexes (a non-empty list). :num_indices_to_merge controls index deltas to merge; zero only indexes new rows, nil/omitted uses the SDK default. :retrain rebuilds vector models (default false) and cannot be combined with :num_indices_to_merge. Missing indexes are not created. Use LanceDB.Index.stats/2 to inspect coverage.

stats(table)

@spec stats(table()) :: result(map())

Returns table statistics as a map with string keys.

Includes num_rows, num_indices, total_bytes, and fragment_stats with fragment counts and physical row-count distributions (min, max, integer mean, p25, p50, p75, p99). In LanceDB 0.37.1, small fragments have fewer than 100,000 physical rows. Physical counts can include deleted rows.

total_bytes sums recorded data, index, and overlay file sizes for this table version. It excludes manifests, deletion files, historical versions, and files whose sizes were not recorded. It is not a complete directory or bucket size. Uses the table handle's version and read-consistency configuration.

uses_v2_manifest_paths(table)

@spec uses_v2_manifest_paths(table()) :: result(boolean())

Returns whether the table uses V2 manifest paths.