barrel_embed_splade (barrel_embed v2.3.1)
View SourceSPLADE sparse embedding provider
Uses SPLADE (Sparse Lexical and Expansion) models for neural sparse embeddings. Produces sparse vectors suitable for inverted index search.
Dependencies (transformers, torch) are installed automatically in the managed venv on first use.
Configuration
Config = #{
model => "prithivida/Splade_PP_en_v1", %% Model name (default)
python => "python3", %% Python executable (default)
timeout => 120000 %% Timeout in ms (default)
}.Sparse Vector Format
Unlike dense embeddings, SPLADE produces sparse vectors:
#{indices => [1, 5, 10], values => [0.5, 0.3, 0.8]}Where indices are vocabulary token IDs and values are weights.
Supported Models
- "prithivida/Splade_PP_en_v1" - Default, SPLADE++ English - "naver/splade-cocondenser-ensembledistil" - NAVER's SPLADE
Use Cases
- Lexical-semantic hybrid search - Term expansion (captures synonyms and related terms) - Efficient inverted index storage
Summary
Functions
Check if provider is available.
Get dimension (vocab size) for this provider. For sparse vectors, dimension is the vocabulary size.
Generate dense embedding (converts sparse to dense). Note: This is inefficient for large vocab sizes. Use embed_sparse/2 instead.
Generate dense embeddings for batch (converts sparse to dense).
Generate sparse embeddings for multiple texts.
Generate sparse embedding for a single text.
Initialize the provider.
Provider name.
Types
-type sparse_vector() :: #{indices := [non_neg_integer()], values := [float()]}.
Functions
Check if provider is available.
-spec dimension(map()) -> pos_integer().
Get dimension (vocab size) for this provider. For sparse vectors, dimension is the vocabulary size.
Generate dense embedding (converts sparse to dense). Note: This is inefficient for large vocab sizes. Use embed_sparse/2 instead.
Generate dense embeddings for batch (converts sparse to dense).
-spec embed_batch_sparse([binary()], map()) -> {ok, [sparse_vector()]} | {error, term()}.
Generate sparse embeddings for multiple texts.
-spec embed_sparse(binary(), map()) -> {ok, sparse_vector()} | {error, term()}.
Generate sparse embedding for a single text.
Initialize the provider.
-spec name() -> atom().
Provider name.