barrel_record_indexer (barrel v1.0.0)

View Source

Record indexer: drives vector indexing from the embed outbox.

One indexer runs per record-mode database. Each round it folds a batch of pending outbox entries, re-reads the CURRENT state of each document (never the state captured in the entry), applies the embedding policy, embeds through barrel's barrel_embed state, writes vectors with the index-only vectordb path, deletes vectors of deleted or non-matching documents, and acks the processed entries by exact HLC key.

Exactly-once effect without a checkpoint: un-acked entries ARE the checkpoint, processing is an idempotent upsert, and a rewrite during processing lands at a new HLC key that re-drives the consumer.

Failure handling:

  • Embed failures count per document id; after ?MAX_FAILURES the id is skipped in later rounds (its entry stays pending and visible) and logged. A batch embed failure falls back to per-item embeds so one poison document cannot fail the batch.
  • vectordb call failures are transient (store restarting): the entries stay un-acked and the round retries later.

Scheduling: a nudge cast (sent by barrel after tagged writes) triggers an immediate round; otherwise the indexer polls with backoff (?POLL_MIN up to ?POLL_MAX when idle).

Summary

Functions

Trigger an immediate indexing round.

Start an indexer. Config: #{name := binary(), db := binary(), vstore := binary(), policy := policy(), embed := term(), batch_size => pos_integer()}.

The indexer's pid for a database, or undefined.

Functions

nudge(Name)

-spec nudge(atom() | binary()) -> ok.

Trigger an immediate indexing round.

start_link(Config)

-spec start_link(map()) -> {ok, pid()} | {error, term()}.

Start an indexer. Config: #{name := binary(), db := binary(), vstore := binary(), policy := policy(), embed := term(), batch_size => pos_integer()}.

whereis_pid(Name)

-spec whereis_pid(atom() | binary()) -> pid() | undefined.

The indexer's pid for a database, or undefined.