barrel (barrel v1.0.0)
View SourceBarrel: the embeddable edge database.
Composes the document layer (barrel_docdb) and the vector layer (barrel_vectordb) behind one API. A barrel database is a docdb database plus a vectordb store that share a name and a single id space: a document, its attachments (blobs), and its vector are all addressed by the same id. Blobs are docdb attachments; their storage backend is pluggable per database via the docdb barrel_att_backend seam.
open/2 returns a handle used by the rest of this module. Each underlying application stays usable on its own; barrel adds no storage of its own, it only coordinates the layers.
Summary
Types
Handle for a composed barrel database. Treat it as opaque: names are binaries internally (atoms are accepted at every API entry and normalized, so dynamic database names never grow the atom table). The optional embedding, embed, and dimensions fields are present on record-mode databases (opened with the embedding option): the validated policy, barrel's barrel_embed state, and the resolved vector dimension. encryption carries the spec the database was opened with (branches inherit it).
Opaque attachment read/write stream handle.
Functions
Abort an attachment writer.
Get attachment metadata (content type, length, digest).
Fork a composed database into a branch (timeline). The docdb forks instantly (at now, or rewound to a past HLC with at => HlcT); the branch opens as its own composed database with a FRESH vector store. Plain databases do not carry vectors across the fork in v1 (they live only in the vector store; re-add or reindex on the branch). Record-mode branches backfill from the embeddings stored in doc bodies, see barrel_record_backfill.
Get changes since an HLC timestamp (first for all).
Get changes since an HLC timestamp, with options.
Close a composed barrel database (indexer, then vector store, then document database).
Close an attachment reader.
Delete a composed barrel database: stop the indexer, destroy the vector store (handles closed, directory removed), and delete the document database's files.
Delete a document attachment.
Delete a document by id. On record-mode databases the delete is tagged so the indexer removes the document's vector; in sync mode the vector is removed before returning.
Delete a document by id with options (e.g. provenance).
Delete multiple documents by id. docdb has no bulk delete, so this maps over delete_doc/2 and returns a result per id, in order.
Every retained version of a document: the current winner (with last-writer provenance when recorded) followed by live conflict siblings and superseded versions.
Explain a BQL query: source, streamability, warnings, and the engine's index strategy for collection queries.
Run a declarative query.
Run a declarative query with options.
Finalise an attachment writer.
Fetch a document attachment.
Get a document by id.
Get a document by id with options.
Get multiple documents by id in one batch. Returns a result per input id, in order.
Get multiple documents by id in one batch, with options.
The retained history log, oldest first: one entry per applied write within the retention window (see barrel_history). Entries carry hlc, id, version, deleted, cause, vv, and provenance when the write recorded it.
Like history/1 with options: from/to (HLC bounds), limit (bounds the SCAN, not the matches), and id (keep only one document's entries; applied while scanning, so combined with limit fewer than limit entries can return).
The oldest HLC the history is complete from (undefined until a retention sweep runs).
Decode a cursor produced by hlc_encode/1 back to an HLC timestamp usable as the Since argument of changes/2.
Encode an HLC timestamp (the cursor returned by changes/2) as a JSON/URL-safe string, for transports that serialise the changes feed.
Database metadata.
List a document's attachment names.
Merge a branch's edits back into its parent (see barrel_docdb:merge_branch/2). When the parent is a record-mode database open in this VM, its indexer is nudged so merged docs embed promptly (the outbox entries exist either way).
Open a composed barrel database with default options.
Open a composed barrel database.
Open a streaming reader for an attachment.
Open a streaming writer for an attachment.
Store a document attachment.
Create or update a document. On record-mode databases the write is tagged for async vector indexing (same signature, policy applied transparently).
Create or update a document with options. On record-mode databases: vector => Vector supplies an explicit embedding (skips the embedder, indexed synchronously); a policy in sync mode embeds before returning (read-your-write search). Both fail the put on embed/dimension errors BEFORE writing; failures after the document committed are healed by the indexer.
Create or update multiple documents in one batch. Returns a result per input document, in order.
Create or update multiple documents in one batch, with options. On a sync-mode record database the batch embeds (embed_batch) before writing and indexes before returning; an embed failure fails the whole batch with nothing written.
Run a BQL query (see barrel_bql). Unlike barrel_docdb:query/2, table-function sources (vector_top_k, bm25_top_k, hybrid_top_k) execute here, joined back to their documents. SUBSCRIBE queries need subscribe_query (live queries).
Run a BQL query with options: params (a map for $name placeholders), chunk_size / continuation (streamable queries), and overfetch (table functions, default 3).
Fold BQL rows without materializing the full result. Fun(Row, Acc) -> {ok, Acc} | {stop, Acc}.
Read the next chunk from an attachment reader.
Semantic search over the vector store.
BM25 keyword search.
Hybrid (vector + BM25) search. On record-mode databases barrel embeds the query itself and passes it as query_vector (the store has no embedder).
Vector search with an explicit query vector.
Subscribe to a changes stream from Since. Returns a stream pid.
Subscribe to a changes stream with options.
Acknowledge a push-mode batch (drives backpressure).
Start a live query for a BQL SUBSCRIBE statement. The initial snapshot and add/change/remove deltas are pushed to the owner process (default: the caller); see barrel_bql_live for the message shapes. Returns an opaque subscription for unsubscribe_query/1; monitor its pid for crash signals. Owner death tears the query down.
Like subscribe_query/2 with options: params, owner.
Stop a changes stream started with subscribe/2,3.
Stop a live query. Idempotent.
Add a document to the vector store (text embedded by the store).
Add a document to the vector store with an explicit vector.
Add many documents to the vector store in one atomic batch.
Delete a vector entry by id.
Get a stored vector entry by id.
Vector store statistics.
The body of one version of a document (current or archived; swept bodies are gone).
Write a chunk to an attachment writer.
Types
-type db() :: #{name := binary(), docdb := binary(), vstore := binary(), embedding => barrel_embedding_policy:policy(), embed => term(), dimensions => pos_integer(), encryption => barrel_keyprovider:spec()}.
Handle for a composed barrel database. Treat it as opaque: names are binaries internally (atoms are accepted at every API entry and normalized, so dynamic database names never grow the atom table). The optional embedding, embed, and dimensions fields are present on record-mode databases (opened with the embedding option): the validated policy, barrel's barrel_embed state, and the resolved vector dimension. encryption carries the spec the database was opened with (branches inherit it).
-type stream() :: term().
Opaque attachment read/write stream handle.
Functions
-spec abort_attachment(stream()) -> ok.
Abort an attachment writer.
Get attachment metadata (content type, length, digest).
Fork a composed database into a branch (timeline). The docdb forks instantly (at now, or rewound to a past HLC with at => HlcT); the branch opens as its own composed database with a FRESH vector store. Plain databases do not carry vectors across the fork in v1 (they live only in the vector store; re-add or reindex on the branch). Record-mode branches backfill from the embeddings stored in doc bodies, see barrel_record_backfill.
Opts: at (now | HlcT), docdb (branch docdb overrides), vectordb (branch vector store config, e.g. db_path), backfill (record mode: sync | none).
Get changes since an HLC timestamp (first for all).
Get changes since an HLC timestamp, with options.
Close a composed barrel database (indexer, then vector store, then document database).
-spec close_attachment_reader(stream()) -> ok.
Close an attachment reader.
Delete a composed barrel database: stop the indexer, destroy the vector store (handles closed, directory removed), and delete the document database's files.
Delete a document attachment.
Delete a document by id. On record-mode databases the delete is tagged so the indexer removes the document's vector; in sync mode the vector is removed before returning.
Delete a document by id with options (e.g. provenance).
Delete multiple documents by id. docdb has no bulk delete, so this maps over delete_doc/2 and returns a result per id, in order.
Every retained version of a document: the current winner (with last-writer provenance when recorded) followed by live conflict siblings and superseded versions.
Explain a BQL query: source, streamability, warnings, and the engine's index strategy for collection queries.
Run a declarative query.
Run a declarative query with options.
Finalise an attachment writer.
Fetch a document attachment.
Get a document by id.
Get a document by id with options.
Get multiple documents by id in one batch. Returns a result per input id, in order.
Get multiple documents by id in one batch, with options.
-spec history(db()) -> {ok, [barrel_history:entry()]} | {error, term()}.
The retained history log, oldest first: one entry per applied write within the retention window (see barrel_history). Entries carry hlc, id, version, deleted, cause, vv, and provenance when the write recorded it.
-spec history(db(), map()) -> {ok, [barrel_history:entry()]} | {error, term()}.
Like history/1 with options: from/to (HLC bounds), limit (bounds the SCAN, not the matches), and id (keep only one document's entries; applied while scanning, so combined with limit fewer than limit entries can return).
-spec history_floor(db()) -> barrel_hlc:timestamp() | undefined.
The oldest HLC the history is complete from (undefined until a retention sweep runs).
Decode a cursor produced by hlc_encode/1 back to an HLC timestamp usable as the Since argument of changes/2.
Encode an HLC timestamp (the cursor returned by changes/2) as a JSON/URL-safe string, for transports that serialise the changes feed.
Database metadata.
List a document's attachment names.
Merge a branch's edits back into its parent (see barrel_docdb:merge_branch/2). When the parent is a record-mode database open in this VM, its indexer is nudged so merged docs embed promptly (the outbox entries exist either way).
Open a composed barrel database with default options.
Open a composed barrel database.
Opts may carry docdb => map() (passed to barrel_docdb:create_db/2, including att_opts to choose an attachment backend) and vectordb => map() (a barrel_vectordb store config; its name is set from Name). Opens the docdb database if it exists, otherwise creates it, then starts the vectordb store.
embedding => PolicyMap opens the database in RECORD MODE: writes are tagged for asynchronous vector indexing driven by the database's embedding policy (see barrel_embedding_policy), the vector store keeps only vectors and indexes (text/metadata read through the record's documents), and BM25 defaults to the disk backend so it survives restarts. Without the option, behavior is unchanged.
encryption => disabled | default | #{provider => Mod} encrypts the WHOLE logical database at rest under one key: the docdb stores take it as-is, and barrel resolves the key once (on the docdb keyspace, so a branch resolves its parent's key) and hands it to the vector store. Runtime config: pass it again on every open. Branches inherit the spec from the parent handle.
Open a streaming reader for an attachment.
-spec open_attachment_writer(db(), binary(), binary(), binary()) -> {ok, stream()} | {error, term()}.
Open a streaming writer for an attachment.
Store a document attachment.
Create or update a document. On record-mode databases the write is tagged for async vector indexing (same signature, policy applied transparently).
Create or update a document with options. On record-mode databases: vector => Vector supplies an explicit embedding (skips the embedder, indexed synchronously); a policy in sync mode embeds before returning (read-your-write search). Both fail the put on embed/dimension errors BEFORE writing; failures after the document committed are healed by the indexer.
Create or update multiple documents in one batch. Returns a result per input document, in order.
Create or update multiple documents in one batch, with options. On a sync-mode record database the batch embeds (embed_batch) before writing and indexes before returning; an embed failure fails the whole batch with nothing written.
Run a BQL query (see barrel_bql). Unlike barrel_docdb:query/2, table-function sources (vector_top_k, bm25_top_k, hybrid_top_k) execute here, joined back to their documents. SUBSCRIBE queries need subscribe_query (live queries).
Run a BQL query with options: params (a map for $name placeholders), chunk_size / continuation (streamable queries), and overfetch (table functions, default 3).
-spec query_fold(db(), binary() | string(), map(), fun((map(), term()) -> {ok, term()} | {stop, term()}), term()) -> {ok, term(), map()} | {error, term()}.
Fold BQL rows without materializing the full result. Fun(Row, Acc) -> {ok, Acc} | {stop, Acc}.
Read the next chunk from an attachment reader.
Semantic search over the vector store.
BM25 keyword search.
Hybrid (vector + BM25) search. On record-mode databases barrel embeds the query itself and passes it as query_vector (the store has no embedder).
Vector search with an explicit query vector.
Subscribe to a changes stream from Since. Returns a stream pid.
Subscribe to a changes stream with options.
Acknowledge a push-mode batch (drives backpressure).
-spec subscribe_query(db(), binary() | string()) -> {ok, #{ref := reference(), pid := pid()}} | {error, term()}.
Start a live query for a BQL SUBSCRIBE statement. The initial snapshot and add/change/remove deltas are pushed to the owner process (default: the caller); see barrel_bql_live for the message shapes. Returns an opaque subscription for unsubscribe_query/1; monitor its pid for crash signals. Owner death tears the query down.
-spec subscribe_query(db(), binary() | string(), map()) -> {ok, #{ref := reference(), pid := pid()}} | {error, term()}.
Like subscribe_query/2 with options: params, owner.
-spec subscribe_stop(pid()) -> ok.
Stop a changes stream started with subscribe/2,3.
Stop a live query. Idempotent.
Add a document to the vector store (text embedded by the store).
Add a document to the vector store with an explicit vector.
Add many documents to the vector store in one atomic batch.
Each element is either {Id, Text, Metadata} (text embedded by the store) or {Id, Text, Metadata, Vector} (explicit vector). All elements must be the same shape; a mixed batch returns {error, mixed_batch}.
Delete a vector entry by id.
Get a stored vector entry by id.
Vector store statistics.
The body of one version of a document (current or archived; swept bodies are gone).
Write a chunk to an attachment writer.