barrel_record_indexer (barrel v1.0.1)
View SourceRecord indexer: drives vector indexing from the embed outbox.
One indexer runs per record-mode database. Each round it folds a batch of pending outbox entries, re-reads the CURRENT state of each document (never the state captured in the entry), applies the embedding policy, embeds through barrel's barrel_embed state, writes vectors with the index-only vectordb path, deletes vectors of deleted or non-matching documents, and acks the processed entries by exact HLC key.
Exactly-once effect without a checkpoint: un-acked entries ARE the checkpoint, processing is an idempotent upsert, and a rewrite during processing lands at a new HLC key that re-drives the consumer.
Failure handling:
- Embed failures count per document id; after ?MAX_FAILURES the id is skipped in later rounds (its entry stays pending and visible) and logged. A batch embed failure falls back to per-item embeds so one poison document cannot fail the batch.
- vectordb call failures are transient (store restarting): the entries stay un-acked and the round retries later.
Scheduling: a nudge cast (sent by barrel after tagged writes) triggers an immediate round; otherwise the indexer polls with backoff (?POLL_MIN up to ?POLL_MAX when idle).
Summary
Functions
Trigger an immediate indexing round.
Start an indexer. Config: #{name := binary(), db := binary(), vstore := binary(), policy := policy(), embed := term(), batch_size => pos_integer()}.
The indexer's pid for a database, or undefined.
Functions
Trigger an immediate indexing round.
Start an indexer. Config: #{name := binary(), db := binary(), vstore := binary(), policy := policy(), embed := term(), batch_size => pos_integer()}.
The indexer's pid for a database, or undefined.