SPLADE Provider

View Source

Sparse embedding generation using SPLADE models.

What is SPLADE?

SPLADE (Sparse Lexical and Expansion) produces sparse vectors instead of dense vectors. Each dimension corresponds to a vocabulary token, enabling:

  • Lexical matching (like BM25)
  • Semantic expansion (synonyms, related terms)
  • Efficient inverted index storage

Requirements

# Using virtualenv with uv (recommended)
./scripts/setup_venv.sh
uv pip install transformers torch --python .venv/bin/python

# Or install manually
pip install transformers torch

Configuration

{ok, State} = barrel_embed:init(#{
    embedder => {splade, #{
        python => "python3",                        % default, fallback only
        model => "prithivida/Splade_PP_en_v1",     % default
        timeout => 120000                           % default, ms
    }}
}).

Options

OptionTypeDefaultDescription
pythonstring"python3"Python executable, used only if the managed venv could not be created
modelstring"prithivida/Splade_PP_en_v1"Model name
timeoutinteger120000Timeout in milliseconds

barrel_embed manages its own Python virtualenv automatically and installs transformers and torch into it on first use. See Python Virtualenv Setup for the managed venv API.

Supported Models

ModelNotes
prithivida/Splade_PP_en_v1Default, SPLADE++
naver/splade-cocondenser-ensembledistilNAVER's SPLADE

Sparse Vector Format

Unlike dense embeddings, SPLADE returns sparse vectors:

#{
    indices => [1, 5, 10, 42, ...],   % vocabulary token IDs
    values => [0.5, 0.3, 0.8, 0.2, ...]  % weights
}

API

Native Sparse API

%% Single text
{ok, SparseVec} = barrel_embed_splade:embed_sparse(<<"query text">>, Config).
#{indices := Indices, values := Values} = SparseVec.

%% Batch
{ok, SparseVecs} = barrel_embed_splade:embed_batch_sparse(Texts, Config).

Dense API (Compatibility)

For compatibility with dense search, SPLADE can convert to dense vectors:

%% Returns dense vector (sparse converted to dense)
{ok, DenseVec} = barrel_embed:embed(<<"text">>, State).

Warning

Dense conversion is memory-intensive for large vocabularies (~30k dimensions). Use native sparse API when possible.

Combine SPLADE with dense embeddings for hybrid search:

%% SPLADE for lexical matching
{ok, SpladeState} = barrel_embed:init(#{embedder => {splade, #{}}}).
{ok, SparseVec} = barrel_embed_splade:embed_sparse(Query, SpladeConfig).

%% Dense for semantic matching
{ok, DenseState} = barrel_embed:init(#{embedder => {ollama, #{...}}}).
{ok, DenseVec} = barrel_embed:embed(Query, DenseState).

%% Combine scores (application-specific)
FinalScore = Alpha * SparseScore + (1 - Alpha) * DenseScore.

Use Cases

  • Keyword search with semantic expansion: SPLADE expands queries with related terms
  • Hybrid retrieval: Combine with dense embeddings
  • Efficient storage: Sparse vectors compress well