barrel_ngram (barrel_ngram v0.7.1)

View Source

barrel_ngram: exact substring search over barrel_docdb.

A byte-level trigram index giving exact lexical recall (identifiers, error strings, config keys) that semantic search misses. A corpus is bound to a database and a gram selector; indexing is driven by the database's changes feed, and every query result is confirmed against the real document text.

M1 usage

   ok = barrel_ngram:open(<<"code">>, #{db => <<"mydb">>}),
   {ok, _} = barrel_ngram:index(<<"code">>),
   {ok, Hits} = barrel_ngram:search(<<"code">>, <<"connect_timeout">>, #{}).

Requires the barrel_ngram application to be started.

Summary

Types

corpus/0

Functions

close(Corpus)

Close a corpus, stopping every shard.

compact(Corpus)

Compact every shard's live segments, physically evicting superseded and deleted entries. Returns {error, busy} if a background compaction is already running on a shard.

index(Corpus)

Catch the corpus up to the current head of its database's changes feed and freeze the buffer. The index is kept live in the background by a feed subscription; this is the synchronous catch-up point for tests and ops. Alias of refresh/1.

is_open(Corpus)

Whether a corpus is currently open (cheap metadata check).

open(Corpus, Opts)

Create or re-attach a corpus bound to a database.

refresh(Corpus)

Synchronously drain the changes feed up to now and freeze every shard's buffer into a segment.

regex(Corpus, Regex)

Equivalent to regex(Corpus, Regex, #{}).

regex(Corpus, Regex, Opts)

Regex search (PCRE syntax). Returns hits with the matching id and the match spans within its corpus text. {error, {bad_regex, _}} if the pattern does not compile.

search(Corpus, Literal)

Equivalent to search(Corpus, Literal, #{}).

search(Corpus, Literal, Opts)

Substring search. Returns hits with the matching document id and the match spans within its corpus text.

Types

corpus/0

-type corpus() :: binary() | atom().

Functions

close(Corpus)

-spec close(corpus()) -> ok.

Close a corpus, stopping every shard.

compact(Corpus)

-spec compact(corpus()) -> {ok, map()} | {error, term()}.

Compact every shard's live segments, physically evicting superseded and deleted entries. Returns {error, busy} if a background compaction is already running on a shard.

index(Corpus)

-spec index(corpus()) -> {ok, map()} | {error, term()}.

Catch the corpus up to the current head of its database's changes feed and freeze the buffer. The index is kept live in the background by a feed subscription; this is the synchronous catch-up point for tests and ops. Alias of refresh/1.

is_open(Corpus)

-spec is_open(corpus()) -> boolean().

Whether a corpus is currently open (cheap metadata check).

open(Corpus, Opts)

-spec open(corpus(), map()) -> ok | {error, term()}.

Create or re-attach a corpus bound to a database.

There is no separate create step: this creates the corpus if it does not exist and re-attaches (resuming from its on-disk state) if it does. It starts a feed subscription that keeps the index in sync. selector, shards, and postings are fixed for the life of a corpus.

Options:

  • db (required) - the barrel_docdb database name to index.
  • selector - gram selector module (default barrel_ngram_selector_dense).
  • selector_opts - selector tuning map (default #{}), e.g. the sparse selector's radius and sample_rate.
  • fields - all or a list of field names to index (default all).
  • shards - number of shards to spread the corpus across by rendezvous hashing (default 1).
  • postings - posting-list codec, varint (default) or roaring (a native bitmap AND for large dense corpora).
  • data_dir - segment storage directory (default from app env); segments live under data_dir/<corpus>/.
  • freeze_threshold - buffer size before an automatic freeze (default 1000).
  • compact_threshold - live segment count before an automatic compaction (default 16; infinity disables it).

refresh(Corpus)

-spec refresh(corpus()) -> {ok, map()} | {error, term()}.

Synchronously drain the changes feed up to now and freeze every shard's buffer into a segment.

regex(Corpus, Regex)

-spec regex(corpus(), binary()) -> {ok, [barrel_ngram_query:hit()]} | {error, term()}.

Equivalent to regex(Corpus, Regex, #{}).

regex(Corpus, Regex, Opts)

-spec regex(corpus(), binary(), map()) -> {ok, [barrel_ngram_query:hit()]} | {error, term()}.

Regex search (PCRE syntax). Returns hits with the matching id and the match spans within its corpus text. {error, {bad_regex, _}} if the pattern does not compile.

search(Corpus, Literal)

-spec search(corpus(), binary()) -> {ok, [barrel_ngram_query:hit()]} | {error, term()}.

Equivalent to search(Corpus, Literal, #{}).

search(Corpus, Literal, Opts)

-spec search(corpus(), binary(), map()) -> {ok, [barrel_ngram_query:hit()]} | {error, term()}.

Substring search. Returns hits with the matching document id and the match spans within its corpus text.