Full-text and hybrid search

Back to README.

These examples use db and table handles. See the API guide for setup. Match the table schema to each example.

Full-text and hybrid search

Create a full-text index on a string column, then search for words:

{:ok, :created} = LanceDB.Index.create(table, :text, type: :fts, name: "text_fts")

{:ok, matches} =
  LanceDB.Query.full_text_search(table, "elixir concurrency",
    text_column: :text,
    columns: [:id, :text],
    filter: "category = 'code'",
    limit: 10
  )

Results include selected fields and "_score", a BM25 relevance score. Higher scores rank first. If no rows match, the result is {:ok, []}. :text_column accepts a string or atom. Omit it to use the SDK's indexed column selection. Set it explicitly when the table has multiple searchable text columns.

:columns and :filter work as in ordinary queries. The default limit is 10 rows.

Indexed FTS uses the index's saved analyzer settings. The default analyzer applies simple tokenization, lowercase conversion, English stemming, stop-word removal, and ASCII folding. Unindexed scans use SDK defaults. Plain strings are term queries. They are not SQL or a general query language. Use LanceDB.FTS for structured queries, as described below. Set analyzer options during index creation.

Text indexes support the same naming, replacement, inspection, deletion, and maintenance operations as other indexes. An explicit :text_column also permits unindexed scans. An FTS index speeds up repeated searches. Searches can find new rows before index optimization.

Combine keyword matching and vector similarity with hybrid search:

{:ok, matches} =
  LanceDB.Query.hybrid_search(table, [0.9, 0.1, 0.2], "elixir concurrency",
    vector_column: :vector,
    text_column: :text,
    columns: [:id, :text],
    distance: :cosine,
    filter: "category = 'code'",
    limit: 10,
    rrf_k: 60
  )

Hybrid search supports indexed and unindexed columns. Specify :text_column for an unindexed text scan. It accepts all vector search options, including :nprobes, :refine_factor, and :bypass_vector_index. It also accepts :text_column and :rrf_k. Supply your own query embedding. A SQL filter applies to both channels.

A row can match either channel. If one channel returns no matches, the other channel can still return results.

By default, the SDK combines ranked candidates with reciprocal rank fusion (RRF). :rrf_k defaults to 60. It must be a positive number that float32 can represent. The SDK uses ranks that start at zero. Each channel supplies 1 / (rrf_k + rank) for a candidate. Fusion adds these values.

Each channel fetches up to limit + offset candidates. After fusion, the query skips the requested offset and returns at most limit results. Fusion cannot find candidates outside those lists.

Results include selected data fields and "_relevance_score". Higher scores rank first. Results do not include individual channel scores. with_row_id: true keeps internal row IDs. Without that option, results do not include row IDs. The fused score combines ranks from both channels.

RRF scores indicate rank. They are not probabilities. Invalid options raise ArgumentError. Native search failures return {:error, reason}.

Custom hybrid ranking and ordering

Hybrid search accepts reranker: :rrf by default. It also accepts a two-argument Elixir function or a module that implements LanceDB.Reranker. A callback receives vector candidates and text candidates as lists of maps. Both lists include "_rowid" and the requested columns or computed projections. Vector rows have "_distance". Text rows have "_score".

normalize: :score normalizes each channel's scores directly. This is the default. normalize: :rank first replaces scores with ranks, including tied ranks. It then normalizes those ranks. Smaller distances stay better. Larger text scores stay better.

This uses the current SDK's float32 normalization, including its handling of almost constant scores. Candidate row order stays unchanged. RRF uses row positions, so this option does not change RRF results.

For example, combine normalized vector similarity and text relevance:

weighted = fn vector_rows, text_rows ->
  scores = Map.new(vector_rows, fn row ->
    {row["_rowid"], 0.7 * (1 - row["_distance"])}
  end)

  scores = Enum.reduce(text_rows, scores, fn row, scores ->
    contribution = 0.3 * row["_score"]
    Map.update(scores, row["_rowid"], contribution, &(&1 + contribution))
  end)

  scores
  |> Enum.map(fn {id, score} -> %{_rowid: id, _relevance_score: score} end)
  |> Enum.sort_by(&{-&1._relevance_score, &1._rowid})
end

{:ok, rows} = LanceDB.Query.hybrid_search(table, vector, "elixir",
  reranker: weighted,
  normalize: :score,
  limit: 20
)

Return scored row-ID maps directly or as {:ok, maps}. {:error, reason} becomes a query error. IDs must come from either input channel. Each ID can occur at most once in output. Scores must be finite numbers that float32 can represent. The operation ignores additional map fields.

Original candidate data supplies result rows. A callback can remove candidates or return an empty list. The result keeps callback order unless you set order_by. Results include _relevance_score. Final results include _rowid only with with_row_id: true.

Callbacks run in a separate process. Callback failures become error tuples. The query timeout includes candidate collection, callback execution, and final processing. A timeout or caller termination cancels a running callback. :rrf_k applies only to :rrf. It cannot be combined with a custom reranker.

Plan and schema inspection do not call callbacks. Analysis executes the callback and reports the final row count.

Ordering and null placement

order_by works with scans, vector searches, full-text searches, hybrid searches, and their streams. Vector searches include batch and multivector searches:

{:ok, rows} = LanceDB.Query.search(table, vector,
  order_by: [desc_nulls_last: :priority, asc: :id]
)

{:ok, rows} = LanceDB.Query.hybrid_search(table, vector, "elixir",
  order_by: [asc_nulls_first: :published_on, desc: :_relevance_score]
)

:asc and :desc keep the default nulls-last behavior. Use :asc_nulls_first, :asc_nulls_last, :desc_nulls_first, or :desc_nulls_last to set null placement for each key. Ordering uses stored columns or query ranking metadata. This also applies when columns does not include the sort column or a computed select alias uses its name. Use a unique final key to resolve ties consistently.

Searches fetch up to limit + offset candidates before ordering and pagination. Ordering cannot find rows outside that candidate set. Batch searches apply this rule separately to each query vector. Hybrid channels first keep their own relevance order. Fusion or the custom callback then runs. Next, order_by sorts the combined results.

Finally, the query applies offset and limit. A different candidate count can change rankings and page contents.

Hybrid streams collect both candidate sets before they return results. A custom callback runs once when the caller reads the stream. A new stream read calls it again. Final results stay in a native cursor. Elixir receives batches with row counts limited by batch_size. Each channel still uses up to limit + offset candidates.

Schema inspection and explain plans stay available. Invalid callback results raise LanceDB.Error when the caller reads the stream.

Structured full-text queries

Use LanceDB.FTS builders where the API accepts search text:

alias LanceDB.FTS
query = FTS.match("elixir concurrency", column: :text, operator: :and)
{:ok, matches} = LanceDB.Query.full_text_search(table, query, columns: [:id, :text])
fuzzy = FTS.match("elixr", column: :text, fuzziness: 1, max_expansions: 50)
{:ok, matches} = LanceDB.Query.full_text_search(table, fuzzy)

query =
  FTS.boolean(
    must: [FTS.match("elixir", column: :text)],
    should: [FTS.match("concurrency", column: :title, boost: 2)],
    must_not: [FTS.match("deprecated", column: :text)]
  )

{:ok, matches} = LanceDB.Query.hybrid_search(table, [0.9, 0.1, 0.2], query)
{:ok, plan} = LanceDB.Query.explain(LanceDB.Query.new(table, {:text, query}, []), verbose: false)

Builders return LanceDB.FTS query values directly. Invalid arguments raise ArgumentError. Database and index errors return {:error, reason} when the query executes. Existing string queries keep their behavior. Structured queries support projection, filters, limits, offsets, row IDs, fast search, and timeouts.

A leaf's :column overrides the search call's default :text_column. Without either option, automatic column selection works only if exactly one text column has an FTS index. Otherwise, specify columns explicitly. Multi-match always uses its named columns. Boolean clause lists accept structured children and plain strings. Plain strings use exact OR term queries.

Phrase example:

{:ok, :created} =
  LanceDB.Index.create(table, :text,
    type: :fts,
    name: "text_fts",
    replace: true,
    analyzer: [with_position: true]
  )

{:ok, matches} =
  LanceDB.Query.full_text_search(table, FTS.phrase("elixir concurrency", slop: 1),
    text_column: :text
  )

Rebuild an existing index with positions if necessary. Phrase queries search indexed positions only. New appends become searchable after Maintenance.optimize_indices/1. Updates can make some index segments unusable for phrase queries. Maintenance restores usable positions. Exact match queries can scan unindexed text with an explicit column.

Fuzzy expansion uses the index vocabulary. It can miss terms that occur only in unindexed data.

Fuzzy matching uses simple query tokenization in this SDK. This can differ from the index's configured normalization. :prefix_length limits leading ASCII characters. Use ASCII query text if the value is positive. A value of zero also accepts Unicode. The package does not expose automatic fuzziness.

LanceDB 0.37.1 treats that setting as exact matching. Set edit distances and expansion limits explicitly.

Token inspection

Inspect analyzer output without a table, or use an existing index's saved settings:

{:ok, tokens} = LanceDB.Tokenizer.tokenize("Running in cafés")
# [%{"text" => "run", "position" => 0}, %{"text" => "cafe", "position" => 2}]

{:ok, tokens} =
  LanceDB.Tokenizer.tokenize("MiXeD café",
    tokenizer: :raw,
    lower_case: false,
    stem: false,
    remove_stop_words: false,
    ascii_folding: false
  )

{:ok, tokens} = LanceDB.Tokenizer.tokenize(table, "Running in cafés", index: "text_fts")
{:ok, tokens} = LanceDB.Tokenizer.tokenize(table, "Running in cafés", column: :text)

tokenize/2 accepts the same analyzer keyword list as index creation. For tokenize/3, set exactly one selector: :index or :column. The selector accepts a string or atom. Column selection works only if the column has exactly one FTS index. Missing, ambiguous, and non-FTS indexes return errors. Inspection reads saved metadata.

This includes the selected historical snapshot when the handle has a historical checkout. Inspection does not change the table or index.

Results use string keys "text" and "position". Positions start at zero. Stop-word removal can leave gaps between positions. Tokens show the configured analyzer output. They do not show complete query execution or fuzzy vocabulary expansion. Results do not include original character or byte offsets.

The function accepts empty input. With the default analyzer, it returns {:ok, []}. Invalid UTF-8 and invalid options raise ArgumentError.

Configurable FTS analyzers

Configure tokenization and normalization when building an FTS index:

{:ok, :created} =
  LanceDB.Index.create(table, :text,
    type: :fts,
    name: "text_fts",
    analyzer: [
      tokenizer: :simple,
      language: "English",
      lower_case: true,
      stem: true,
      remove_stop_words: false,
      ascii_folding: true
    ]
  )

analyzer: accepts a keyword list only for type: :fts. If you do not set it, the SDK defaults apply. The index stores these settings. Keyword and hybrid queries use them, even after you reopen the table.

Boolean options must be true or false. The simple analyzer defaults to lowercase conversion, stemming, stop-word removal, and ASCII folding. Positions default to false. The code tokenizer has separate SDK defaults. For n-gram settings, use tokenizer: :ngram. N-grams cannot store positions.

The package does not include tokenizers that use external language models.

For case-sensitive exact token matching, use :raw. Disable lowercase conversion, stemming, stop-word removal, and ASCII folding. For n-gram matching, consider disabling stemming and stop-word removal. Invalid option structures raise ArgumentError. An unsupported language returns an SDK error.

FTS index creation also accepts block_size: 128 | 256, positive memory_limit_mb, positive num_workers, and format_version: :v1 | :v2 | :v3. These build settings are separate from analyzer:. If no format is supplied, the SDK selects it. Use V3 for blocks with 256 documents or for code tokenization. The SDK treats V3 as experimental. Memory limits apply to index construction.

They do not limit total process memory.

For JSON document analysis, store valid JSON strings and construct scalar matches with FTS.json_match/3:

{:ok, :created} = LanceDB.Index.create(table, :text, type: :fts,
  analyzer: [document_tokenizer: :json])
query = LanceDB.FTS.json_match("metadata.language", "elixir", column: :text)
{:ok, rows} = LanceDB.Query.full_text_search(table, query, wand_factor: 1.0)

JSON queries accept strings, numbers, booleans, and nil. Dotted paths select nested values. Arrays use their containing path. The current tokenizer cannot escape commas or semicolons in paths. It also cannot escape semicolons in values. The helper rejects these characters and unsupported fuzzy matching.

For token inspection with a JSON analyzer, use path,type,value syntax. For example, use title,str,elixir.

Full-text and hybrid queries accept wand_factor. It must be nil or a finite float32-compatible number greater than or equal to 1. A value of 1 keeps exact WAND pruning. Larger values permit faster top-k search with lower recall. Builders, streams, and query diagnostics keep this setting. It does not apply to scans or vector-only queries.

To change an analyzer, rebuild its index with the same name and replace: true. Different options alone do not change an existing index. Unindexed scans have no saved analyzer settings and use SDK defaults.