LanceDB.Index (LanceDB v0.2.0)

Copy Markdown View Source

Index creation, inspection, prewarming and removal.

Summary

Functions

Creates an index on one column, returning {:ok, :created} after completion.

Starts native index creation and returns a Job.

Drops an index by name. Physical files may remain until database cleanup.

Lists Index.Description structs with name, type, columns, size and indexed row counts.

Builds a reusable, validated index definition from keyword options.

Loads an existing index into the index cache, returning after prewarming completes. Useful before the first search after opening a table. Does not build or update indexes; cached pages can later be evicted. Missing indexes return an error. For shared background warming, use LanceDB.TableCache.prewarm/3.

Returns index statistics as a string-keyed map, or {:ok, nil} if absent. Includes "loss": the sum of IVF training losses across index segments, or nil for non-vector indexes or when any segment does not report loss. Loss is not a query recall measurement.

Waits until every named index exists and has zero unindexed rows.

Types

result(value)

@type result(value) :: LanceDB.result(value)

table()

@type table() :: LanceDB.Table.t()

Functions

create(table, column, opts \\ [])

@spec create(table(), atom() | String.t(), keyword() | LanceDB.Index.Definition.t()) ::
  result(:created)

Creates an index on one column, returning {:ok, :created} after completion.

Types: :auto (default), :btree, :bitmap, :label_list, :fm, :ivf_flat, :ivf_pq, :ivf_sq, :ivf_rq, :hnsw_flat, :hnsw_pq, :hnsw_sq, :fts. :label_list accelerates array_has_any/array_has_all on lists; :fm accelerates contains(column, 'substring') on strings. Neither is selected by :auto; choose its type explicitly. Options: :name, :replace (default false), and, for vector indexes, :distance (:l2, :cosine, :dot, or :hamming for uint8 vectors), :num_partitions or :target_partition_size (mutually exclusive), :sample_rate (SDK default 256), and :max_iterations (SDK default 50). Counts must be positive uint32 integers. :ivf_pq and :hnsw_pq also accept :num_sub_vectors. HNSW types accept :m (graph neighbors, SDK default 20) and :ef_construction (build candidates, SDK default 300). All HNSW types use IVF partitions. PQ types accept :num_bits of 4 or 8 (default 8); 4-bit PQ needs an even number of subvectors. :ivf_rq accepts 1 through 9 bits (default 1), and its vector dimension must be divisible by 8. SQ is fixed at 8 bits. :fts accepts an :analyzer keyword list (tokenizer, language, normalization, positions, token length, and n-gram settings, custom_stop_words, document_tokenizer (:text/:json), and code splitting flags; see the analyzer guide). FTS build controls are :block_size (128/256), positive uint32 :memory_limit_mb and :num_workers, and :format_version (:v1, :v2, :v3). Code tokenization and 256-document blocks require V3; omitted format selects it automatically when needed. Explicit scalar indexes accept train: false to create an empty index. Vector indexes require training data; their distance must match the distance used for search.

create_async(table, column, opts \\ [])

@spec create_async(
  table(),
  atom() | String.t(),
  keyword() | LanceDB.Index.Definition.t()
) ::
  result(LanceDB.Job.t())

Starts native index creation and returns a Job.

drop(table, name)

@spec drop(table(), atom() | String.t()) :: result(:dropped)

Drops an index by name. Physical files may remain until database cleanup.

list(table)

@spec list(table()) :: result([LanceDB.Index.Description.t()])

Lists Index.Description structs with name, type, columns, size and indexed row counts.

new(opts \\ [])

Builds a reusable, validated index definition from keyword options.

prewarm(table, name)

@spec prewarm(table(), atom() | String.t()) :: result(:prewarmed)

Loads an existing index into the index cache, returning after prewarming completes. Useful before the first search after opening a table. Does not build or update indexes; cached pages can later be evicted. Missing indexes return an error. For shared background warming, use LanceDB.TableCache.prewarm/3.

stats(table, name)

@spec stats(table(), atom() | String.t()) :: result(map() | nil)

Returns index statistics as a string-keyed map, or {:ok, nil} if absent. Includes "loss": the sum of IVF training losses across index segments, or nil for non-vector indexes or when any segment does not report loss. Loss is not a query recall measurement.

wait(table, names, opts \\ [])

@spec wait(table(), atom() | String.t() | [atom() | String.t()], keyword()) ::
  result(:ready)

Waits until every named index exists and has zero unindexed rows.

Accepts a name or non-empty list of names. Options: :timeout_ms (default 60_000, maximum two hours) and :poll_interval_ms (default 1_000, maximum 60_000). Both must be positive integers. Returns {:ok, :ready} or a timeout Error containing details.pending_indexes. Other inspection errors propagate immediately.

This follows the SDK's readiness semantics: each name must be observed ready once. Concurrent writes or index replacement can invalidate that observation. The table handle's read-consistency settings apply to changes by other handles. Waiting neither builds nor optimizes indexes, and a timeout does not cancel any index jobs. Use Job.wait/2 to await one specific creation job instead. Polling sleeps in the caller without holding native admission capacity.