PostgreSQL pgvector-backed vector store.

This is the default vector store backend, using the existing Arcana schema with pgvector extension for similarity search.

Breaking name change

In v1.7 the hybrid-search opts :semantic_weight and :fulltext_weight were renamed to :vector_weight and :keyword_weight. The old names are no longer accepted: callers passing them get a Logger.warning and the default 0.5 weight.

Configuration

config :arcana, vector_store: :pgvector  # default

How hybrid blends the two scores

:hybrid mode scores each chunk on cosine similarity and on ts_rank, then blends them by :vector_weight and :keyword_weight. Two details decide what the keyword half is worth.

A chunk only scores on the keyword side if it actually satisfies the query. plainto_tsquery builds an AND query, but ts_rank scores term overlap regardless, so a chunk carrying some of the terms and not others gets a real score while matching nothing - and a term-dense one can out-score a chunk that genuinely answers the query. Keyword mode has always gated on @@; hybrid does too.

The scores are then scaled against the best keyword hit in the candidate set, so results stay comparable across queries. On its own that maps the best hit to 1.0 however weak it is, which turns "nothing really matched" into a full-strength signal. :keyword_score_floor is the ts_rank treated as full strength: when the whole set falls below it, scores are scaled against the floor instead, so a weak set stays weak.

# a corpus whose genuine matches score lower than most
Arcana.search("question", mode: :hybrid, keyword_score_floor: 0.02)

# or globally
config :arcana, search: [keyword_score_floor: 0.02]

It defaults to 0.05. Raise it if lexical noise still promotes wrong chunks, lower it if real matches in your corpus are being damped, and set 0 to scale against the set's own best the way earlier versions did. ts_rank magnitudes depend on document length and term frequency, so the useful value is corpus-specific.

Tuning recall with :hnsw_ef_search

An HNSW index scan considers hnsw.ef_search candidates and only then applies the query's filters, so a search scoped to a collection or a source can come back with fewer rows than exist - occasionally none. pgvector's default is 40. Raising it finds more of the true nearest neighbours, at the cost of more work per query:

Arcana.search("question", collections: ["docs"], hnsw_ef_search: 200)

It can also be a global search default:

config :arcana, search: [hnsw_ef_search: 200]

Three things worth knowing. It only applies when the planner actually chooses an index scan - on a small table a sequential scan is exact and the setting is irrelevant. Because hnsw.ef_search is a GUC, applying it means running the search inside a transaction; if you already have one open, the setting stays in effect for the rest of your transaction rather than just the search.

And it costs a round-trip. Setting it adds a transaction and one set_config query per search, and Arcana.search/2 searches each collection separately - so a query over three collections pays it three times, and as a global default it applies to every search. That is deliberate rather than hoisted up to wrap the whole retrieval: retrieval also embeds the query, and holding a database transaction open across a call to an embedding service is worse than an extra round-trip. Set it per call on the searches that need the recall, rather than globally, if that matters to you.

Applies to the chunk search in :vector and :hybrid modes. :keyword never touches the vector index, so it ignores the option.

It does not reach the graph paths. Arcana.Graph.GraphStore.Ecto's entity search runs its own filtered query against arcana_graph_entities' HNSW index, so with graph: true (or Arcana.ask/2's graph context) the chunk search honours the option while the entity match can still under-return. Extending it there is a separate change.

Notes

This backend works with the existing arcana_chunks and arcana_documents tables. The collection parameter maps to the document's collection_id.

For simpler use cases without the full document schema, consider the :memory backend.