barrel_vectordb_turboquant (barrel_vectordb v2.1.2)
View SourceTurboQuant: Data-oblivious 3-bit vector quantization
TurboQuant provides efficient vector compression without training. Based on Google Research's algorithm combining:
1. PolarQuant: Random rotation + polar coordinate conversion 2. QJL: 1-bit Johnson-Lindenstrauss error correction
Key advantages over Product Quantization (PQ): - No training required (data-oblivious) - Deterministic with seed (reproducible results) - ~8x compression for 768-dim vectors (vs 4x for 8-bit scalar) - Only 1-3% recall loss vs float32
Storage format (D=768, 3-bit): Header: 4 bytes (version, bits, dimension flags) PolarQuant: ceil(D*bits/8) bytes QJL: ceil(D/8) bytes Total: ~388 bytes vs 3072 bytes float32
Summary
Functions
Compute ADC distance for multiple codes using SIMD-accelerated NIF. Amortizes NIF call overhead for batch operations.
Batch encode multiple vectors
Decode TurboQuant code back to approximate vector
Compute asymmetric distance using precomputed tables Full ADC with correct polar distance formula: d^2 = r_q^2 + r_d^2 - 2*r_q*r_d*cos(theta_diff) = QRSq + r_d^2 - CosTerm * r_d
Compute ADC distance using SIMD-accelerated NIF. This is the optimized version that should be used for production workloads.
Encode a vector using TurboQuant Returns compact binary representation
Get configuration info
Create a new TurboQuant configuration (no training needed) Options: bits - bits per component (default: 3, range: 2-4) qjl_bits - QJL error correction bits (default: 1) dimension - vector dimension (required, must be even) seed - random seed for rotation matrix (default: 42) qjl_iterations - QJL correction iterations (default: 5) qjl_learning_rate - QJL gradient learning rate (default: 0.1)
Precompute distance lookup tables for a query vector This enables fast asymmetric distance computation (ADC)
Types
-type distance_tables() :: binary().
-type tq_code() :: binary().
-type tq_config() :: #tq_config{bits :: 2..4, qjl_bits :: pos_integer(), dimension :: pos_integer(), rotation_seed :: integer(), rotation_matrix :: binary(), qjl_matrix :: binary(), qjl_dim :: pos_integer(), angle_levels :: [float()], qjl_iterations :: non_neg_integer(), qjl_learning_rate :: float()}.
Functions
-spec batch_distance_nif(distance_tables(), [tq_code()]) -> [float()].
Compute ADC distance for multiple codes using SIMD-accelerated NIF. Amortizes NIF call overhead for batch operations.
Batch encode multiple vectors
Decode TurboQuant code back to approximate vector
-spec distance(distance_tables(), tq_code()) -> float().
Compute asymmetric distance using precomputed tables Full ADC with correct polar distance formula: d^2 = r_q^2 + r_d^2 - 2*r_q*r_d*cos(theta_diff) = QRSq + r_d^2 - CosTerm * r_d
Performance note: This ADC computation can be moved to C/NIF for better performance if needed, similar to barrel_vectordb_hnsw distance functions. The tight loop over pairs with table lookups would benefit from SIMD.
-spec distance_nif(distance_tables(), tq_code()) -> float().
Compute ADC distance using SIMD-accelerated NIF. This is the optimized version that should be used for production workloads.
Encode a vector using TurboQuant Returns compact binary representation
Get configuration info
Create a new TurboQuant configuration (no training needed) Options: bits - bits per component (default: 3, range: 2-4) qjl_bits - QJL error correction bits (default: 1) dimension - vector dimension (required, must be even) seed - random seed for rotation matrix (default: 42) qjl_iterations - QJL correction iterations (default: 5) qjl_learning_rate - QJL gradient learning rate (default: 0.1)
-spec precompute_tables(tq_config(), [float()]) -> distance_tables().
Precompute distance lookup tables for a query vector This enables fast asymmetric distance computation (ADC)
Full ADC table format per pair: QRSq (32-bit float) + NumLevels * CosTerm (32-bit floats) where CosTerm_i = 2 * r_q * cos(theta_q - angle_center_i)
Distance formula: d^2 = r_q^2 + r_d^2 - 2*r_q*r_d*cos(theta_diff) = QRSq + r_d^2 - CosTerm * r_d