Getting started

barrel_ngram gives you exact substring and regex search over the documents in a barrel_docdb database. It is the lexical counterpart to semantic search: it finds identifiers, error strings, config keys, and punctuation-heavy literals that embeddings miss. Use it when you need to find documents that literally contain a string or match a pattern. This page takes you from an empty database to your first search.

Requirements

The barrel_ngram application must be started (it runs a supervision subtree). Under a release it starts with barrel_ngram in the boot order; in a shell or a test:

{ok, _} = application:ensure_all_started(barrel_ngram).

Create a corpus

A corpus is a named index bound to one database. You create it with barrel_ngram:open/2: there is no separate "create" step, open/2 creates the corpus if it does not exist and re-attaches to it (resuming from its on-disk state) if it does. Opening it starts a background subscription to the database's changes feed, so the index stays in sync as documents are written.

The name is any binary or atom and is yours to choose; a common convention is the database name.

%% index the database <<"mydb">> under a corpus named <<"code">>
ok = barrel_ngram:open(<<"code">>, #{db => <<"mydb">>}).

db is the only required option. The rest tune the index:

Some examples:

%% a sharded, roaring corpus for a large code database
ok = barrel_ngram:open(<<"code">>,
                       #{db => <<"repo">>,
                         shards => 8,
                         postings => roaring,
                         data_dir => "/var/lib/barrel/ngram"}),

%% index only two fields of each document
ok = barrel_ngram:open(<<"notes">>,
                       #{db => <<"mydb">>, fields => [<<"title">>, <<"body">>]}).

The corpus persists on disk under data_dir/<corpus>/. After a restart, calling open/2 again with the same name and data_dir re-attaches and resumes from where it left off (it replays only the feed tail since its last commit). Its options (db, phase2_selector_opts, fields, shards, postings) are fixed for the life of the corpus: reopening with a different value fails with {error, {config_mismatch, Field, Persisted, Requested}} rather than silently reindexing, rebinding to a different database, or orphaning the old shard set. To change one, open a new corpus under a different name or data_dir and let it reindex.

A corpus name is a filesystem path component: it must not be empty, ., .., or contain /, \, or a NUL byte -- open/2 rejects an unsafe name with {error, {invalid_option, corpus, Corpus}} before touching anything on disk.

open/2 and close/1 for the same corpus never run concurrently; a second call for a corpus already mid-open/close waits for the first to finish rather than racing it.

Index and search

Writes reach the index through the feed subscription. For a deterministic point where the index has caught up (tests, ops), call refresh/1, then search:

{ok, _Summary} = barrel_ngram:refresh(<<"code">>),

%% substring search: every document whose text contains the literal
{ok, Hits} = barrel_ngram:search(<<"code">>, <<"connect_timeout">>).
%% Hits = [#{id => <<"doc-a">>, spans => [{7, 15}]}, ...]

Each hit carries the document id and the byte spans where the literal occurs in the document's indexed text.

Regex search

{ok, More} = barrel_ngram:regex(<<"code">>, <<"connect_\\w+timeout">>).

Regex uses PCRE syntax (it compiles with re). A malformed pattern returns {error, {bad_regex, Reason}}. See regex for what accelerates.

Case-insensitive search

Both search and regex take case_sensitive => false as a third-argument option (default true):

{ok, Hits} = barrel_ngram:search(<<"code">>, <<"connect_timeout">>, #{case_sensitive => false}),
{ok, More} = barrel_ngram:regex(<<"code">>, <<"error">>, #{case_sensitive => false}).

See regex for the ASCII/non-ASCII split and the errors a non-ASCII pattern or a non-UTF-8 document can return.

Close a corpus

ok = barrel_ngram:close(<<"code">>).

close/1 returns ok | {error, term()}. It is idempotent: closing a corpus that is already closed, or was never opened, is ok.

Notes

• index/1 is an alias of refresh/1.
• The corpus indexes the binary string values of each document's non-reserved top-level fields (see selectors and design for what gets indexed).
• Results are always exact: the trigram index only narrows candidates, and a confirm pass re-checks each candidate against the current document.
• A corpus indexed by a version of barrel_ngram older than 0.9.0 has no corpus.meta on disk; open/2 rejects it with {error, {legacy_corpus_requires_reindex, Corpus}} since its db/shards binding was never recoverable. Reindex it into a fresh data_dir.