Local embedding provider using Bumblebee and Nx.Serving.
Uses HuggingFace models locally. Default is BAAI/bge-small-en-v1.5 (384 dimensions).
Configuration
# Default model
config :arcana, embedder: :local
# Custom HuggingFace model and serving options
config :arcana,
embedder: {:local,
model: "BAAI/bge-large-en-v1.5",
batch_size: 64,
sequence_length: 512,
batch_timeout: 100
}Starting the Serving
Add Arcana.Embedder.Local to your application supervision tree. It will
read its configuration from the global :arcana, :embedder config:
children = [
Arcana.Embedder.Local,
# ... other children
]Or pass options inline (these override the global config):
children = [
{Arcana.Embedder.Local, model: "BAAI/bge-small-en-v1.5"},
# ... other children
]Options
:model- HuggingFace model ID (default:BAAI/bge-small-en-v1.5):batch_size- Nx.Serving batch size (default: 32). Larger batches give better GPU throughput but increase memory usage.:sequence_length- Max token length per input (default: 512):batch_timeout- Milliseconds to wait collecting requests into a batch (default: 100). Lower values reduce latency, higher values improve throughput.
Summary
Functions
Returns the child spec for starting the embedding serving.
Prepares text for embedding by adding model-specific prefixes.
Starts the Nx.Serving for this embedder.
Functions
Returns the child spec for starting the embedding serving.
Prepares text for embedding by adding model-specific prefixes.
E5 models require query: prefix for search queries and passage: prefix
for documents. Other models return text unchanged.
Options
:query- Text is a search query (adds "query: " prefix for E5):document- Text is document content (adds "passage: " prefix for E5)nil- Defaults to:documentfor E5 models
Examples
iex> prepare_text("hello", "intfloat/e5-small-v2", :query)
"query: hello"
iex> prepare_text("hello", "intfloat/e5-small-v2", :document)
"passage: hello"
iex> prepare_text("hello", "BAAI/bge-small-en-v1.5", :query)
"hello"Starts the Nx.Serving for this embedder.
Reads options from the global :arcana, :embedder config when no opts
are passed (or merges when some opts are passed).