barrel_vectordb_pq (barrel_vectordb v2.1.2)
View SourceProduct Quantization for DiskANN
Product Quantization (PQ) compresses vectors by: 1. Splitting D-dimensional vector into M subspaces 2. Training K centroids per subspace (k-means) 3. Encoding vector as M bytes (indices into codebooks)
Distance computation uses precomputed lookup tables: - For query q, precompute distance from q's subvectors to all centroids - Distance to any encoded vector = sum of M table lookups
Memory: M bytes per vector (vs D*4 for float32) With M=8, K=256: ~32x compression for 256-dim vectors
Summary
Functions
Batch encode multiple vectors
Decode PQ code back to approximate vector
Compute asymmetric distance using precomputed tables This is the fast O(M) lookup vs O(D) for full distance Tables size must be M*K*4 where M = byte_size(Code)
Encode a vector to M bytes using trained codebooks
Get PQ configuration info
Check if PQ is trained
Create a new PQ configuration Options: m - number of subquantizers (default: 8) k - centroids per subquantizer (default: 256, max: 256) dimension - vector dimension (required, must be divisible by m)
Precompute distance tables for a query vector Returns M*K float32 values: distance from each query subvector to each centroid
Train codebooks using k-means clustering Vectors should be a list of float lists, all same dimension
Types
-type distance_tables() :: binary().
M*K float32 values
-type pq_code() :: binary().
M bytes
-type pq_config() :: #pq_config{m :: pos_integer(), k :: pos_integer(), dimension :: pos_integer(), subvector_dim :: pos_integer(), codebooks :: [binary()] | undefined, trained :: boolean()}.
Functions
Batch encode multiple vectors
Decode PQ code back to approximate vector
-spec distance(distance_tables(), pq_code()) -> float().
Compute asymmetric distance using precomputed tables This is the fast O(M) lookup vs O(D) for full distance Tables size must be M*K*4 where M = byte_size(Code)
Encode a vector to M bytes using trained codebooks
Get PQ configuration info
Check if PQ is trained
Create a new PQ configuration Options: m - number of subquantizers (default: 8) k - centroids per subquantizer (default: 256, max: 256) dimension - vector dimension (required, must be divisible by m)
-spec precompute_tables(pq_config(), [float()]) -> distance_tables().
Precompute distance tables for a query vector Returns M*K float32 values: distance from each query subvector to each centroid
Train codebooks using k-means clustering Vectors should be a list of float lists, all same dimension