barrel_faiss (barrel_faiss v1.0.0)

Erlang bindings for FAISS vector similarity search.

FAISS (Facebook AI Similarity Search) is a library for efficient similarity search and clustering of dense vectors.

Quick Start

  %% Create a flat index for 128-dimensional vectors
  {ok, Index} = barrel_faiss:new(128),
 
  %% Add vectors (binary of packed float32)
  Vectors = <<1.0:32/float-native, 2.0:32/float-native, ...>>,
  ok = barrel_faiss:add(Index, Vectors),
 
  %% Search for nearest neighbors
  {ok, Distances, Labels} = barrel_faiss:search(Index, Query, 10),
 
  %% Clean up
  ok = barrel_faiss:close(Index).

Vector Format

Vectors are passed as binaries of packed 32-bit floats in native endianness. For n vectors of dimension d, the binary size is n * d * 4 bytes.

Index Types

Use index_factory/2 to create different index types:

<<"Flat">> - Exact search, no training needed
<<"HNSW32">> - Fast approximate search, no training
<<"IVF100,Flat">> - Inverted file index, requires training
<<"IVF100,PQ8">> - IVF with product quantization

Summary

Types

index/0

metric_type/0

Functions

add(Index, Vectors)

Add vectors to the index.

add_with_ids(Index, Vectors, IDs)

Add vectors with explicit IDs.

close(Index)

Close and release index resources.

deserialize(Binary)

Deserialize index from binary.

dimension(Index)

Get the dimension of vectors in the index.

index_factory(Dimension, Description)

Create an index using a factory string.

index_factory(Dimension, Description, Metric)

Create an index using a factory string with specified metric.

is_trained(Index)

Check if the index is trained.

new(Dimension)

Create a new flat L2 index.

new(Dimension, Metric)

Create a new flat index with specified metric.

ntotal(Index)

Get the number of vectors in the index.

read_index(Path)

Read index from file.

remove_ids(Index, IDs)

Remove vectors by ID.

search(Index, Queries, K)

Search for k nearest neighbors.

serialize(Index)

Serialize index to binary.

train(Index, Vectors)

Train the index with sample vectors.

write_index(Index, Path)

Write index to file.

Types

index/0

-opaque index()

metric_type/0

-type metric_type() :: l2 | inner_product.

Functions

add(Index, Vectors)

-spec add(Index :: index(), Vectors :: binary()) -> ok | {error, term()}.

Add vectors to the index.

Vectors must be a binary of packed float32 values in native endianness. The binary size must be n * dimension * 4 bytes.

This function runs on a dirty CPU scheduler.

Example

  %% Add 3 vectors of dimension 4
  Vectors = <<1.0:32/float-native, 2.0:32/float-native, 3.0:32/float-native, 4.0:32/float-native,
              5.0:32/float-native, 6.0:32/float-native, 7.0:32/float-native, 8.0:32/float-native,
              9.0:32/float-native, 10.0:32/float-native, 11.0:32/float-native, 12.0:32/float-native>>,
  ok = barrel_faiss:add(Index, Vectors).

add_with_ids(Index, Vectors, IDs)

-spec add_with_ids(Index :: index(), Vectors :: binary(), IDs :: binary()) -> ok | {error, term()}.

Add vectors with explicit IDs.

Vectors must be a binary of packed float32 values in native endianness. IDs must be a binary of packed int64 values in native endianness. The number of vectors must match the number of IDs.

Note: Not all index types support add_with_ids. Use IDMap wrapper if needed.

This function runs on a dirty CPU scheduler.

Example

  Vectors = <<1.0:32/float-native, 2.0:32/float-native, 3.0:32/float-native, 4.0:32/float-native>>,
  IDs = <<100:64/signed-native>>,
  ok = barrel_faiss:add_with_ids(Index, Vectors, IDs).

close(Index)

-spec close(Index :: index()) -> ok.

Close and release index resources.

After closing, the index reference becomes invalid.

deserialize(Binary)

-spec deserialize(Binary :: binary()) -> {ok, index()} | {error, term()}.

Deserialize index from binary.

Restores an index previously serialized with serialize/1.

This function runs on a dirty CPU scheduler.

Example

  {ok, Binary} = rocksdb:get(Db, <<"my_index">>),
  {ok, Index} = barrel_faiss:deserialize(Binary).

See also: serialize/1.

dimension(Index)

-spec dimension(Index :: index()) -> pos_integer() | {error, term()}.

Get the dimension of vectors in the index.

index_factory(Dimension, Description)

-spec index_factory(Dimension :: pos_integer(), Description :: binary()) ->
                       {ok, index()} | {error, term()}.

Create an index using a factory string.

Equivalent to index_factory(Dimension, Description, l2).

See also: index_factory/3.

index_factory(Dimension, Description, Metric)

-spec index_factory(Dimension :: pos_integer(), Description :: binary(), Metric :: metric_type()) ->
                       {ok, index()} | {error, term()}.

Create an index using a factory string with specified metric.

The factory string describes the index structure. Common options:

<<"Flat">> - Exact search
<<"HNSW32">> - Hierarchical NSW graph (32 neighbors)
<<"IVF100,Flat">> - IVF with 100 centroids
<<"IVF100,PQ8">> - IVF + product quantization (8 bytes/vector)
<<"PQ16">> - Product quantization only

IVF indexes require training before adding vectors.

Example

  {ok, Index} = barrel_faiss:index_factory(128, <<"HNSW32">>),
  ok = barrel_faiss:add(Index, Vectors).

is_trained(Index)

-spec is_trained(Index :: index()) -> boolean() | {error, term()}.

Check if the index is trained.

Flat and HNSW indexes are always trained. IVF indexes require explicit training with train/2.

new(Dimension)

-spec new(Dimension :: pos_integer()) -> {ok, index()} | {error, term()}.

Create a new flat L2 index.

Equivalent to new(Dimension, l2).

See also: new/2.

new(Dimension, Metric)

-spec new(Dimension :: pos_integer(), Metric :: metric_type()) -> {ok, index()} | {error, term()}.

Create a new flat index with specified metric.

Creates an IndexFlat which performs exact (brute-force) search. No training is required.

Metrics

l2 - Euclidean distance (smaller = more similar)
inner_product - Inner product (larger = more similar)

Example

  {ok, Index} = barrel_faiss:new(128, l2),
  true = barrel_faiss:is_trained(Index).

ntotal(Index)

-spec ntotal(Index :: index()) -> non_neg_integer() | {error, term()}.

Get the number of vectors in the index.

read_index(Path)

-spec read_index(Path :: binary()) -> {ok, index()} | {error, term()}.

Read index from file.

This function runs on a dirty IO scheduler.

remove_ids(Index, IDs)

-spec remove_ids(Index :: index(), IDs :: binary()) ->
                    {ok, Removed :: non_neg_integer()} | {error, term()}.

Remove vectors by ID.

IDs must be a binary of packed int64 values in native endianness. Returns the number of vectors actually removed.

Note: Not all index types support removal. HNSW indexes will return an error.

This function runs on a dirty CPU scheduler.

Example

  %% Remove vectors with IDs 0, 1, 2
  IDs = <<0:64/signed-native, 1:64/signed-native, 2:64/signed-native>>,
  {ok, Removed} = barrel_faiss:remove_ids(Index, IDs),
  3 = Removed.

search(Index, Queries, K)

-spec search(Index :: index(), Queries :: binary(), K :: pos_integer()) ->
                {ok, Distances :: binary(), Labels :: binary()} | {error, term()}.

Search for k nearest neighbors.

Queries must be a binary of packed float32 values. Returns distances and labels as binaries.

This function runs on a dirty CPU scheduler.

Return Format

Distances - nq * k float32 values
Labels - nq * k int64 values (-1 if fewer than k results)

Example

  Query = <<1.5:32/float-native, 2.5:32/float-native, 3.5:32/float-native, 4.5:32/float-native>>,
  {ok, Distances, Labels} = barrel_faiss:search(Index, Query, 5),
  DistList = [D || <<D:32/float-native>> <= Distances],
  LabelList = [L || <<L:64/signed-native>> <= Labels].

serialize(Index)

-spec serialize(Index :: index()) -> {ok, binary()} | {error, term()}.

Serialize index to binary.

The returned binary can be stored in any K/V database and later restored with deserialize/1.

This function runs on a dirty CPU scheduler.

Example

  {ok, Binary} = barrel_faiss:serialize(Index),
  ok = rocksdb:put(Db, <<"my_index">>, Binary).

See also: deserialize/1.

train(Index, Vectors)

-spec train(Index :: index(), Vectors :: binary()) -> ok | {error, term()}.

Train the index with sample vectors.

Required for IVF indexes before adding vectors. The number of training vectors should be at least nlist * 39 where nlist is the number of centroids (e.g., 100 for IVF100).

This function runs on a dirty CPU scheduler.

Example

  {ok, Index} = barrel_faiss:index_factory(128, <<"IVF100,Flat">>),
  false = barrel_faiss:is_trained(Index),
  ok = barrel_faiss:train(Index, TrainingVectors),
  true = barrel_faiss:is_trained(Index).

write_index(Index, Path)

-spec write_index(Index :: index(), Path :: binary()) -> ok | {error, term()}.

Write index to file.

This function runs on a dirty IO scheduler.