# Design notes & non-goals Where the design stands on what is **not built yet** — the reasoning, the trade-offs, and what was **refused**. This is not a roadmap: no dates, no commitments. It records *how we think about* each direction so the decisions don't get relitigated; some of these may never ship. What orders the list: cover what the global search of a business application actually needs — without a separate search service to run, and with an index that lives in the same transaction as your data. --- ## Async indexing `indexing_strategy :sync | :notify | :manual` on `SearchAsh.Source`, with **no hard Oban dependency**: `:notify` emits an Ash notification, `:manual` lets the host drive a durable job while keeping the DSL, `Document.to_attrs/2` and `reindex/2`. Today indexing is synchronous inside the source write's transaction — which is the whole reason the index cannot drift. The async path trades that guarantee for write latency, so it must be an explicit choice, never a default. --- ## Cross-language search One query probing several languages at once. Each row is pre-stemmed in its own language into a `'simple'` tsvector, so a search currently probes one language at a time. --- ## A `:native` per-row `regconfig` strategy Using Postgres's own stemmers (`to_tsvector('french', …)`) instead of the Elixir pipeline, for the languages Postgres supports. Would cost the per-row-language freedom that motivated `:pre_stemmed` in the first place, so it is an alternative strategy, never a replacement. --- ## Semantic / vector search Vector similarity in Postgres, plus an embedding model to produce the vectors. A different problem with its own cost model — a possible extension point, not a goal. --- ## Refused, with the reason **BM25 relevance ranking.** *(Per-field `weights` shipped in 0.4.0 — this is about going further.)* Its three ideas — inverse document frequency, term-frequency saturation, length normalization — fix the failure modes of *long, heterogeneous* corpora: keyword stuffing, and long documents drowning short precise ones. Business documents are short and structured; nobody keyword-stuffs an invoice. Getting real BM25 in Postgres means a heavy third-party extension, against this project's no-NIF, minimal-dependency line. Weighted fields (`setweight`) cover the practical need better: with structured data you *know* the reference outweighs the description — declaring it is more predictable than discovering it statistically. **Copying authorization into the index (`extra_attrs`).** The rule the whole design follows: > **Derived from content → the index may carry it. Authorization fact → never.** "This document belongs to client 42" changes when the document changes, so the sync re-syncs it. "User 7 may read this document" changes on its own — nothing triggers a re-index, and a stale authorization row is a security incident rather than a cosmetic bug. Filter on content columns and keep the rule in your application; see the authorization section of `SearchAsh.GlobalIndex`. --- ## Known limits, not planned as work - **`ts_rank` has no top-k optimization**: ranking scores every matching row, so cost follows how many rows *match*, not table size. Selective queries stay fast on a large table; a very broad one is where it would be felt. A tenant filter absorbs most of it. - **Composite string primary keys can collide** in `source_id` (`":"`-joined). Fixing it changes the stored format, so it waits for a major version. - **Index creation is not `CONCURRENTLY`** — plan the migration on a large existing table. - **Facet counts (`counts_by/4`) re-run the match per value.** Each value gets its own `Ash.count`, so the full-text match is evaluated N times. This is fine for the low-cardinality columns facets are for (status, type, a handful of tags/months); pass `:values` to bound a wider one. A single `GROUP BY` would be one pass, but it counts *outside* the read action's authorization pipeline and so leaks past the index's policies (measured) — the exact content-vs-authorization boundary this design forbids. Making it safe means reconstructing Ash's authorization filter, which is research, not a quick optim. For genuinely high-cardinality aggregation, that is the boundary where a dedicated search service earns its place.