barrel_ngram (barrel_ngram v0.9.0)
View Sourcebarrel_ngram: exact substring search over barrel_docdb.
A byte-level trigram index giving exact lexical recall (identifiers, error strings, config keys) that semantic search misses. A corpus is bound to a database and a gram selector; indexing is driven by the database's changes feed, and every query result is confirmed against the real document text.
M1 usage
ok = barrel_ngram:open(<<"code">>, #{db => <<"mydb">>}),
{ok, _} = barrel_ngram:index(<<"code">>),
{ok, Hits} = barrel_ngram:search(<<"code">>, <<"connect_timeout">>, #{}).Requires the barrel_ngram application to be started.
A second, positional (phase-2) index narrows candidates to a specific byte position and, with a source configured (see barrel_ngram_source), verifies by reading just that window instead of the whole document. See barrel_ngram_planner's moduledoc for how narrowing and case-insensitive search interact.
open/2 and close/1 are serialized per corpus (never interleaved, even across concurrent callers) by a one-shot barrel_ngram_corpus_lifecycle coordinator -- see its moduledoc for the full lifecycle design.
Summary
Functions
Close a corpus, stopping every shard. Idempotent: closing a corpus that was never durably opened (or is already closed) is ok.
Compact every shard's live segments, physically evicting superseded and deleted entries. Returns {error, busy} if a background compaction is already running on a shard.
Catch the corpus up to the current head of its database's changes feed and freeze the buffer. The index is kept live in the background by a feed subscription; this is the synchronous catch-up point for tests and ops. Alias of refresh/1.
Whether a corpus is currently open: a cheap, NON-AUTHORITATIVE pre-filter, not the safety guarantee (that is safe_shard_call/2, used by every actual query/fan call site -- a shard dying in the instant after this check still returns {error, corpus_not_open} there, never a crash). Checks the declared meta AND that every one of its shard refs currently resolves via whereis_name/1, not just the meta alone -- meta is a persistent_term entry that survives an in-process supervision cascade (e.g. barrel_ngram_registry crashing, which force-restarts barrel_ngram_shard_sup empty via rest_for_one), so meta alone would keep reporting true for every previously-open corpus in the VM even though none of them have a live shard anymore. Only ever reports true once a corpus is fully activated and query-trusted -- never for a corpus still mid-open (interrupted or in-flight) or one whose bookkeeping-only activation write failed.
Create or re-attach a corpus bound to a database.
Synchronously drain the changes feed up to now and freeze every shard's buffer into a segment.
Equivalent to regex(Corpus, Regex, #{}).
Regex search (PCRE syntax). Returns hits with the matching id and the match spans within its corpus text. {error, {bad_regex, _}} if the pattern does not compile.
gen_server:call/3 to a shard, converted from a possible crash/noproc into {error, corpus_not_open} (the shard is gone or shutting down) or {error, {shard_call_failed, Reason}} (a genuine internal error, distinguishable and never silently mislabeled). This is the SAFETY GUARANTEE for a query/fan/lifecycle call racing a shard that dies in the narrow window after any cheap liveness pre-check -- not the pre-check itself. Target is either a bare pid() (when the caller already holds a captured, verified pid -- e.g. the lifecycle coordinator's own existing-shard config diff) or a shard ref() (via the registry, the common query-path case).
Equivalent to search(Corpus, Literal, #{}).
Substring search. Returns hits with the matching document id and the match spans within its corpus text.
Types
Functions
Close a corpus, stopping every shard. Idempotent: closing a corpus that was never durably opened (or is already closed) is ok.
Compact every shard's live segments, physically evicting superseded and deleted entries. Returns {error, busy} if a background compaction is already running on a shard.
Catch the corpus up to the current head of its database's changes feed and freeze the buffer. The index is kept live in the background by a feed subscription; this is the synchronous catch-up point for tests and ops. Alias of refresh/1.
Whether a corpus is currently open: a cheap, NON-AUTHORITATIVE pre-filter, not the safety guarantee (that is safe_shard_call/2, used by every actual query/fan call site -- a shard dying in the instant after this check still returns {error, corpus_not_open} there, never a crash). Checks the declared meta AND that every one of its shard refs currently resolves via whereis_name/1, not just the meta alone -- meta is a persistent_term entry that survives an in-process supervision cascade (e.g. barrel_ngram_registry crashing, which force-restarts barrel_ngram_shard_sup empty via rest_for_one), so meta alone would keep reporting true for every previously-open corpus in the VM even though none of them have a live shard anymore. Only ever reports true once a corpus is fully activated and query-trusted -- never for a corpus still mid-open (interrupted or in-flight) or one whose bookkeeping-only activation write failed.
Create or re-attach a corpus bound to a database.
There is no separate create step: this creates the corpus if it does not exist and re-attaches (resuming from its on-disk state) if it does. It starts a feed subscription that keeps the index in sync. phase2_selector_opts, fields, shards, postings, and db (including which underlying database INSTANCE, not just name -- a delete+recreate under the same name is detected) are fixed for the life of a corpus: reopening with a different value fails with {error, {config_mismatch, Field, Persisted, Requested}} rather than silently reindexing or rebinding under the new value.
Every corpus indexes both a dense (phase-1, exhaustive) and a sparse (phase-2, content-defined, positional) index; there is no longer a corpus-wide selector choice. selector is rejected outright with {error, {unsupported_option, selector}}.
Options:
db(required) - the barrel_docdb database name to index.phase2_selector_opts- phase-2 sampling tuning map (default#{}):radius(0-256) andsample_rate(1 to 2^32).fields-allor a list of field names to index (defaultall).shards- number of shards to spread the corpus across by rendezvous hashing (default 1, max 4096).postings- posting-list codec,varint(default) orroaring(a native bitmap AND for large dense corpora).data_dir- segment storage directory (default from app env); segments live underdata_dir/<corpus>/.freeze_threshold- buffer size before an automatic freeze (default 1000).compact_threshold- live segment count before an automatic compaction (default 16;infinitydisables it).source-{Module, InitArg}, abarrel_ngram_sourcefor verifying candidates without a fullbarrel_docdbfetch (optional; falls back tobarrel_docdb:get_docs/2when absent).
Synchronously drain the changes feed up to now and freeze every shard's buffer into a segment.
-spec regex(corpus(), binary()) -> {ok, [barrel_ngram_query:hit()]} | {error, term()}.
Equivalent to regex(Corpus, Regex, #{}).
-spec regex(corpus(), binary(), map()) -> {ok, [barrel_ngram_query:hit()]} | {error, term()}.
Regex search (PCRE syntax). Returns hits with the matching id and the match spans within its corpus text. {error, {bad_regex, _}} if the pattern does not compile.
-spec safe_shard_call(pid() | barrel_ngram_shards:ref(), term()) -> term() | {error, corpus_not_open} | {error, {shard_call_failed, term()}}.
gen_server:call/3 to a shard, converted from a possible crash/noproc into {error, corpus_not_open} (the shard is gone or shutting down) or {error, {shard_call_failed, Reason}} (a genuine internal error, distinguishable and never silently mislabeled). This is the SAFETY GUARANTEE for a query/fan/lifecycle call racing a shard that dies in the narrow window after any cheap liveness pre-check -- not the pre-check itself. Target is either a bare pid() (when the caller already holds a captured, verified pid -- e.g. the lifecycle coordinator's own existing-shard config diff) or a shard ref() (via the registry, the common query-path case).
-spec search(corpus(), binary()) -> {ok, [barrel_ngram_query:hit()]} | {error, term()}.
Equivalent to search(Corpus, Literal, #{}).
-spec search(corpus(), binary(), map()) -> {ok, [barrel_ngram_query:hit()]} | {error, term()}.
Substring search. Returns hits with the matching document id and the match spans within its corpus text.