barrel_docdb (barrel_docdb v1.0.0)

View Source

barrel_docdb - Public API for barrel_docdb

This module provides the main public API for interacting with barrel_docdb databases. It supports:

  • Database lifecycle management (create, open, close, delete)
  • Document CRUD operations with MVCC revision control
  • Binary attachments stored efficiently with BlobDB
  • Secondary indexes (views) with automatic updates
  • Changes feed for tracking document modifications
  • Replication primitives for syncing databases

Quick Start

   %% Create a database
   {ok, _} = barrel_docdb:create_db(<<"mydb">>),
  
   %% Store a document
   {ok, #{<<"id">> := DocId, <<"rev">> := Rev}} =
       barrel_docdb:put_doc(<<"mydb">>, #{
           <<"id">> => <<"doc1">>,
           <<"type">> => <<"user">>,
           <<"name">> => <<"Alice">>
       }),
  
   %% Retrieve the document
   {ok, Doc} = barrel_docdb:get_doc(<<"mydb">>, <<"doc1">>),
  
   %% Update the document (must include _rev)
   {ok, _} = barrel_docdb:put_doc(<<"mydb">>, Doc#{<<"name">> => <<"Bob">>}),
  
   %% Delete the document
   {ok, _} = barrel_docdb:delete_doc(<<"mydb">>, DocId).

Document Structure

Documents are Erlang maps with the following special keys:

  • <<"id">> - Document identifier (auto-generated if not provided)
  • <<"_rev">> - Revision identifier (managed by the system)
  • <<"_deleted">> - Set to true for deleted documents

Summary

Functions

Abort an attachment writer and clean up partial data.

Attachment feed entries since an HLC (exclusive), in write order. One entry per attachment: op put or delete, digest, length, content_type, seq (feed HLC) and origin (the LWW timestamp). Options: limit.

Oldest HLC the attachment feed is complete from (undefined when never swept or when the backend has no feed).

Fork a database into a new branch (timeline). See barrel_timeline:branch_db/3 for the options.

Close an attachment stream.

Close a database.

Create a new database with default options.

Create a new database with options.

Get database information.

Get the pid of a database by name.

Delete an attachment with options (origin_hlc for replicated deletes: last-write-wins guarded, lands a tombstone).

Delete a database and all its data.

Delete a database, locating a CLOSED database's files under data_dir (the option, else the app env default). Open databases are stopped first, using their actual path.

Delete a document.

Delete a document with options.

Delete a local document.

Delete a global system document.

Digest diff: which of the offered attachments does this database already hold with the same content?

The replication diff: which offered versions does this database not cover? Takes #{DocId => VersionToken}, answers #{DocId => missing | have} (have = the doc's version vector contains the offered version).

Ensure the system database exists. Called automatically by system_doc operations.

Explain a query execution plan.

Find documents matching a query specification.

Find documents with additional options.

Finish writing an attachment and store metadata.

Fold over all documents in the database.

Fold over all documents with options.

Fold over the retained history log in HLC order. Runs in the caller's process. See barrel_history for the entry shape.

Fold over the retained history log with options (from, to, limit).

Fold local documents whose id starts with a prefix. Local docs bypass the path index, so this is the only way to enumerate them (find/2 cannot see them).

Fold over system documents with a given prefix.

Retrieve an attachment.

Get attachment metadata without reading the data.

Get changes since an HLC timestamp.

Get changes with options.

Get list of conflicting revisions for a document.

Get a document by ID.

Get a document with options.

Read a document for replication: current body (tombstones included) plus its version token, encoded version vector, and deleted flag. Caller-side read; the database must be addressed by name.

Every version of a document still resolvable: the current winner plus conflict and superseded siblings.

Get multiple documents by ID (batch read).

Get multiple documents with options (batch read).

Get the current global HLC timestamp.

Get a local document.

Get a global system document.

The body of one version of a document (current or archived).

The oldest HLC the history is complete from (undefined means complete since database creation).

List all attachments for a document.

The OPEN branches of a database (a branch that exists on disk but is not running is not listed).

List all open databases.

Merge a branch's edits back into its parent. See barrel_timeline:merge_branch/2.

Generate a new HLC timestamp.

Return this node's stable identity.

Open a stream for reading an attachment in chunks.

Open a stream for writing an attachment in chunks.

Open an attachment writer with options (origin_hlc, expected_digest: both checked at finish before anything commits).

Open an existing database.

Acknowledge processed outbox entries by their exact HLC keys.

Fold over pending outbox entries for a tag, in HLC order.

Fold over pending outbox entries with options (limit).

Attach binary data to a document.

Store an attachment with options: sync, content_type, origin_hlc (replicated writes; the last-write-wins guard may answer {ok, ignored}) and expected_digest (verified before commit).

Create or update a document.

Create or update a document with options.

Put multiple documents in a single batch.

Put multiple documents with options.

Store a local document.

Store a global system document.

Apply a replicated version (the version-vector protocol write).

Run a BQL query (see barrel_bql) against document indexes.

Read the next chunk from an attachment stream.

Maintenance: synthesize feed rows for attachments written before the feed existed (they do not sync otherwise). Rebuilt rows carry the minimum origin, so any real write wins over them.

Store a computed embedding as a document's embedding column.

Subscribe to document changes matching a path pattern.

Subscribe to document changes with options.

Subscribe to a changes stream.

Subscribe to a changes stream with options.

Subscribe to document changes matching a query.

Subscribe to document changes matching a query with options.

Run a retention sweep now (the database also sweeps on a timer). Returns sweep statistics and the new floor.

Run one doc TTL sweep pass now (see the ttl_sweep_interval database option). Returns the number of docs tombstoned.

Synchronize with a remote HLC timestamp.

Unsubscribe from document change notifications.

Unsubscribe from query-based change notifications.

Validate a database name. Accepts: lowercase alphanumerics, underscore, hyphen. First char must be alphanumeric. Length 1..63. Internal system databases (prefix _) are accepted to support _barrel_system and similar.

Write a chunk of data to an attachment writer.

Types

att_info/0

-type att_info() ::
          #{name := binary(),
            content_type := binary(),
            length := non_neg_integer(),
            digest := binary(),
            chunked => boolean(),
            chunk_size => pos_integer(),
            chunk_count => pos_integer()}.

change/0

-type change() :: map().

db_config/0

-type db_config() :: #{path => string(), store => module(), atom() => term()}.

db_name/0

-type db_name() :: binary().

db_ref/0

-type db_ref() :: pid() | atom().

doc/0

-type doc() :: #{binary() => term()}.

doc_info/0

-type doc_info() :: #{id := docid(), rev := revid(), deleted := boolean(), revtree := revtree()}.

docid/0

-type docid() :: binary().

endpoint/0

-type endpoint() :: db_name() | {node(), db_name()} | {module(), term()}.

rep_options/0

-type rep_options() ::
          #{continuous => boolean(),
            since => seq_string(),
            filter => fun((doc()) -> boolean()),
            atom() => term()}.

rev_info/0

-type rev_info() ::
          #{id := revid(),
            parent := revid() | undefined,
            deleted := boolean(),
            attachments => #{binary() => att_info()}}.

revid/0

-type revid() :: binary().

revtree/0

-type revtree() :: #{revid() => rev_info()}.

seq/0

-type seq() :: barrel_hlc:timestamp().

seq_string/0

-type seq_string() :: binary().

view_name/0

-type view_name() :: binary().

view_result/0

-type view_result() :: #{key := term(), value := term(), id := docid()}.

Functions

abort_attachment_writer(Writer)

-spec abort_attachment_writer(map()) -> ok.

Abort an attachment writer and clean up partial data.

Use this to clean up when an upload fails or is cancelled before finish_attachment_writer/1 is called.

att_changes(Db, Since)

-spec att_changes(binary() | pid(), barrel_hlc:timestamp() | first) ->
                     {ok, [map()], barrel_hlc:timestamp() | first} | {error, term()}.

Attachment feed entries since an HLC (exclusive), in write order. One entry per attachment: op put or delete, digest, length, content_type, seq (feed HLC) and origin (the LWW timestamp). Options: limit.

att_changes(Db, Since, Opts)

-spec att_changes(binary() | pid(), barrel_hlc:timestamp() | first, map()) ->
                     {ok, [map()], barrel_hlc:timestamp() | first} | {error, term()}.

att_floor(Db)

-spec att_floor(binary() | pid()) -> barrel_hlc:timestamp() | undefined | {error, term()}.

Oldest HLC the attachment feed is complete from (undefined when never swept or when the backend has no feed).

branch_db(Parent, BranchName, Opts)

-spec branch_db(binary(), binary(), map()) -> {ok, pid()} | {error, term()}.

Fork a database into a new branch (timeline). See barrel_timeline:branch_db/3 for the options.

close_attachment_stream(Stream)

-spec close_attachment_stream(map()) -> ok.

Close an attachment stream.

close_db(Name)

-spec close_db(binary() | pid()) -> ok | {error, term()}.

Close a database.

Stops the database process and releases resources. The database can be reopened by calling create_db/1 again.

Example

  ok = barrel_docdb:close_db(<<"mydb">>).

create_db(Name)

-spec create_db(binary()) -> {ok, pid()} | {error, term()}.

Create a new database with default options.

Creates a new database with the given name using default settings. The database will be stored in the default data directory.

Example

  {ok, Pid} = barrel_docdb:create_db(<<"mydb">>).

See also: create_db/2.

create_db(Name, Opts)

-spec create_db(binary(), map()) -> {ok, pid()} | {error, term()}.

Create a new database with options.

Creates a new database with custom configuration options.

Options

  • data_dir - Directory to store database files (default: /tmp/barrel_data)
  • store_opts - RocksDB options for document store
  • att_opts - RocksDB options for attachment store
  • encryption - disabled | default | #{provider => Mod}: encrypt both stores at rest (RocksDB EncryptedEnv, AES-256-CTR) with a per-database key resolved by barrel_keyprovider. Runtime config like channels: pass it again on every reopen. A database cannot change between plaintext and encrypted after creation (default: disabled)

Example

  {ok, Pid} = barrel_docdb:create_db(<<"mydb">>, #{
      data_dir => "/var/lib/barrel"
  }).

db_info(Name)

-spec db_info(binary() | pid()) -> {ok, map()} | {error, term()}.

Get database information.

Returns metadata about the database including its name, path, and pid.

Example

  {ok, Info} = barrel_docdb:db_info(<<"mydb">>),
  Name = maps:get(name, Info).

db_pid(Name)

-spec db_pid(binary()) -> {ok, pid()} | {error, not_found}.

Get the pid of a database by name.

Returns the process identifier for the given database name. Useful for operations that require direct access to the database server.

delete_attachment(Db, DocId, AttName)

-spec delete_attachment(binary() | pid(), binary(), binary()) -> ok | {error, term()}.

Delete an attachment.

Removes an attachment from a document.

delete_attachment(Db, DocId, AttName, Opts)

-spec delete_attachment(binary() | pid(), binary(), binary(), map()) -> ok | {error, term()}.

Delete an attachment with options (origin_hlc for replicated deletes: last-write-wins guarded, lands a tombstone).

delete_db(Name)

-spec delete_db(binary()) -> ok | {error, term()}.

Delete a database and all its data.

Permanently removes the database, including all documents, attachments, and indexes. This operation cannot be undone.

Example

  ok = barrel_docdb:delete_db(<<"mydb">>).

delete_db(Name, Opts)

-spec delete_db(binary(), map()) -> ok | {error, term()}.

Delete a database, locating a CLOSED database's files under data_dir (the option, else the app env default). Open databases are stopped first, using their actual path.

delete_doc(Db, DocId)

-spec delete_doc(binary() | pid(), binary()) -> {ok, map()} | {error, term()}.

Delete a document.

Marks a document as deleted. The document's revision history is preserved for conflict resolution and replication.

Example

  {ok, Result} = barrel_docdb:delete_doc(<<"mydb">>, <<"doc1">>).

See also: delete_doc/3.

delete_doc(Db, DocId, Opts)

-spec delete_doc(binary() | pid(), binary(), map()) -> {ok, map()} | {error, term()}.

Delete a document with options.

Options

  • rev - Expected current revision (for conflict detection)

delete_local_doc(Db, DocId)

-spec delete_local_doc(binary() | pid(), binary()) -> ok | {error, not_found}.

Delete a local document.

delete_system_doc(DocId)

-spec delete_system_doc(binary()) -> ok | {error, not_found}.

Delete a global system document.

diff_attachments(Db, Entries)

-spec diff_attachments(binary() | pid(), [#{id := binary(), name := binary(), digest := binary()}]) ->
                          {ok, [#{id := binary(), name := binary(), status := have | missing}]} |
                          {error, term()}.

Digest diff: which of the offered attachments does this database already hold with the same content?

diff_versions(Db, TokenMap)

-spec diff_versions(binary() | pid(), #{binary() => binary()}) ->
                       {ok, #{binary() => missing | have}} | {error, term()}.

The replication diff: which offered versions does this database not cover? Takes #{DocId => VersionToken}, answers #{DocId => missing | have} (have = the doc's version vector contains the offered version).

ensure_system_db()

-spec ensure_system_db() -> ok.

Ensure the system database exists. Called automatically by system_doc operations.

explain(Db, QuerySpec)

-spec explain(binary() | pid(), map()) -> {ok, map()} | {error, term()}.

Explain a query execution plan.

Returns information about how a query would be executed without actually running it. Useful for understanding query performance and optimization.

Example

  {ok, Explanation} = barrel_docdb:explain(<<"mydb">>, #{
      where => [{path, [<<"type">>], <<"user">>}]
  }),
  Strategy = maps:get(strategy, Explanation).
  %% Returns: index_seek | index_scan | multi_index | full_scan

find(Db, QuerySpec)

-spec find(binary() | pid(), map()) -> {ok, [map()], map()} | {error, term()}.

Find documents matching a query specification.

Executes a declarative query against the path index. All document paths are automatically indexed, enabling ad-hoc queries without predefined views.

Note: top-level fields whose key begins with _ (e.g. &lt;&lt;"_meta"&gt;&gt;) are reserved metadata. They are stripped before storage and are neither persisted nor indexed, so they cannot be queried. Use a non-_ top-level namespace for application data.

Query Specification

  • where - List of conditions (required, unless an id scan is used)
  • select - Fields to return (optional, defaults to full doc)
  • order_by - Field or variable to sort by (optional)
  • limit - Maximum results (optional)
  • offset - Skip first N results (optional)
  • include_docs - Include full documents (optional, default true)
  • flat - When true (with include_docs), return flat documents Doc#{&lt;&lt;"id"&gt;&gt;} instead of #{&lt;&lt;"id"&gt;&gt;, &lt;&lt;"doc"&gt;&gt;} wrappers (optional, default false)

Id scans (primary key)

Standalone scans over the document id, ordered, O(matches), without a where clause (the id is not in the path index):

  • id_prefix - binary; all docs whose id starts with the prefix
  • id_range - {Start, End}; docs with Start =&lt; id &lt; End (half-open). Start/End may be undefined for an open bound

Hierarchical/scannable keys should be modelled in the id (e.g. &lt;&lt;"user:123"&gt;&gt;); other fields use where.

Result shape

With include_docs => true (the default), each result is a wrapper #{&lt;&lt;"id"&gt;&gt; => Id, &lt;&lt;"doc"&gt;&gt; => Doc} (use flat => true for the flat document; flat docs carry &lt;&lt;"id"&gt;&gt; but not &lt;&lt;"_rev"&gt;&gt; - use get_doc/2 if the rev is needed). With include_docs => false each result is #{&lt;&lt;"id"&gt;&gt; => Id}.

Conditions

  • {path, Path, Value} - Equality match on path
  • {compare, Path, Op, Value} - Comparison (Op: '>', '<', '>=', '=<', '=/=')
  • {and', [Clauses]}' - All conditions must match
  • {or', [Clauses]}' - Any condition must match
  • {not', Clause}' - Negation
  • {in, Path, Values} - Value in list
  • {contains, Path, Value} - Array contains value
  • {exists, Path} - Path exists
  • {missing, Path} - Path does not exist
  • {regex, Path, Pattern} - Regex match
  • {prefix, Path, Prefix} - String prefix match

Example

  %% Find all active users in org1
  {ok, Results} = barrel_docdb:find(<<"mydb">>, #{
      where => [
          {path, [<<"type">>], <<"user">>},
          {path, [<<"org_id">>], <<"org1">>},
          {path, [<<"status">>], <<"active">>}
      ],
      limit => 100
  }).

See also: explain/2, find/3.

find(Db, QuerySpec, Opts)

-spec find(binary() | pid(), map(), map()) -> {ok, [map()], map()} | {error, term()}.

Find documents with additional options.

Same as find/2 but allows merging additional options into the query. Supports chunked execution with continuation tokens for large result sets.

Example

  %% Basic query
  {ok, Results, Meta} = barrel_docdb:find(<<"mydb">>,
      #{where => [{path, [<<"type">>], <<"user">>}]},
      #{limit => 10, include_docs => false}
  ).
 
  %% Chunked iteration
  {ok, R1, #{has_more := true, continuation := Token}} =
      barrel_docdb:find(Db, Query, #{chunk_size => 100}),
  {ok, R2, #{has_more := false}} =
      barrel_docdb:find(Db, Query, #{continuation => Token}).

finish_attachment_writer(Writer)

-spec finish_attachment_writer(map()) -> {ok, map()} | {ok, ignored} | {error, term()}.

Finish writing an attachment and store metadata.

fold_docs(Db, Fun, Acc)

-spec fold_docs(binary() | pid(), fun((map(), term()) -> {ok, term()} | {stop, term()} | stop), term()) ->
                   {ok, term()}.

Fold over all documents in the database.

Iterates over all non-deleted documents, calling the provided function for each document. The function receives the document and an accumulator.

Callback Return Values

  • {ok, NewAcc} - Continue with new accumulator
  • {stop, FinalAcc} - Stop iteration with final accumulator
  • stop - Stop iteration with current accumulator

Example

  %% Count all documents
  {ok, Count} = barrel_docdb:fold_docs(<<"mydb">>,
      fun(_Doc, Acc) -> {ok, Acc + 1} end,
      0
  ).

fold_docs(Db, Fun, Acc, Opts)

-spec fold_docs(binary() | pid(),
                fun((map(), term()) -> {ok, term()} | {stop, term()} | stop),
                term(),
                map()) ->
                   {ok, term()}.

Fold over all documents with options.

Same as fold_docs/3 but accepts an options map.

Options

- include_deleted: boolean() - include deleted documents (default: false)

fold_history(Db, Fun, Acc)

-spec fold_history(binary(),
                   fun((barrel_history:entry(), term()) -> {ok, term()} | {stop, term()}),
                   term()) ->
                      {ok, term()} | {error, term()}.

Fold over the retained history log in HLC order. Runs in the caller's process. See barrel_history for the entry shape.

fold_history(Db, Fun, Acc, Opts)

-spec fold_history(binary(),
                   fun((barrel_history:entry(), term()) -> {ok, term()} | {stop, term()}),
                   term(),
                   map()) ->
                      {ok, term()} | {error, term()}.

Fold over the retained history log with options (from, to, limit).

fold_local_docs(Db, Prefix, Fun, Acc0)

-spec fold_local_docs(binary() | pid(), binary(), fun((binary(), map(), term()) -> term()), term()) ->
                         {ok, term()} | {error, term()}.

Fold local documents whose id starts with a prefix. Local docs bypass the path index, so this is the only way to enumerate them (find/2 cannot see them).

fold_system_docs(Prefix, Fun, Acc0)

-spec fold_system_docs(binary(), fun((binary(), map(), term()) -> term()), term()) -> {ok, term()}.

Fold over system documents with a given prefix.

Example

  {ok, Federations} = barrel_docdb:fold_system_docs(
      <<"federation:">>,
      fun(_DocId, Doc, Acc) -> [Doc | Acc] end,
      []
  ).

get_attachment(Db, DocId, AttName)

-spec get_attachment(binary() | pid(), binary(), binary()) -> {ok, binary()} | {error, term()}.

Retrieve an attachment.

Gets the binary data of an attachment.

Example

  {ok, Data} = barrel_docdb:get_attachment(<<"mydb">>, <<"doc1">>,
      <<"greeting.txt">>).

get_attachment_info(Db, DocId, AttName)

-spec get_attachment_info(binary() | pid(), binary(), binary()) -> {ok, map()} | {error, term()}.

Get attachment metadata without reading the data.

Useful for checking attachment size, content type, etc. before downloading.

get_changes(Db, Since)

-spec get_changes(binary() | pid(), barrel_hlc:timestamp() | first) ->
                     {ok, [map()], barrel_hlc:timestamp()}.

Get changes since an HLC timestamp.

Returns all document changes since the given HLC timestamp. Use first to get all changes from the beginning.

Example

  %% Get all changes
  {ok, Changes, LastHlc} = barrel_docdb:get_changes(<<"mydb">>, first),
 
  %% Get incremental changes
  {ok, NewChanges, NewHlc} = barrel_docdb:get_changes(<<"mydb">>, LastHlc).

See also: get_changes/3.

get_changes(Db, Since, Opts)

-spec get_changes(binary() | pid(), barrel_hlc:timestamp() | first, map()) ->
                     {ok, [map()], barrel_hlc:timestamp()} | {error, term()}.

Get changes with options.

Options

  • limit - Maximum number of changes to return
  • include_docs - Include full documents in results
  • descending - Reverse order
  • doc_ids - Filter to specific document IDs
  • paths - MQTT-style path patterns (uses the path index)
  • query - Query spec filter over document bodies
  • channel - A configured channel: one bounded scan of the write-time channel feed (not combinable with paths/doc_ids; include_leaves surfaces departure rows tagged left)

get_conflicts(Db, DocId)

-spec get_conflicts(binary() | pid(), binary()) -> {ok, [binary()]} | {error, term()}.

Get list of conflicting revisions for a document.

Returns the list of revision IDs that are in conflict with the current winning revision. An empty list means no conflicts.

Example

  {ok, Conflicts} = barrel_docdb:get_conflicts(<<"mydb">>, <<"doc1">>).
  %% Conflicts = [<<"2-abc">>, <<"2-def">>]

get_doc(Db, DocId)

-spec get_doc(binary() | pid(), binary()) -> {ok, map()} | {error, term()}.

Get a document by ID.

Retrieves a document from the database. Returns {error, not_found} if the document doesn't exist or has been deleted.

Example

  {ok, Doc} = barrel_docdb:get_doc(<<"mydb">>, <<"doc1">>),
  Name = maps:get(<<"name">>, Doc).

See also: get_doc/3.

get_doc(Db, DocId, Opts)

-spec get_doc(binary() | pid(), binary(), map()) ->
                 {ok, map()} | {ok, binary(), map()} | {error, term()}.

Get a document with options.

Retrieves a document with additional options.

Options

  • include_deleted - If true, returns deleted documents
  • rev - Specific revision to retrieve

Example

  %% Get a deleted document
  {ok, Doc} = barrel_docdb:get_doc(<<"mydb">>, <<"doc1">>, #{
      include_deleted => true
  }).

get_doc_for_replication(Db, DocId)

-spec get_doc_for_replication(binary(), binary()) ->
                                 {ok,
                                  #{doc := map(),
                                    version := binary(),
                                    vv := binary(),
                                    deleted := boolean()}} |
                                 {error, term()}.

Read a document for replication: current body (tombstones included) plus its version token, encoded version vector, and deleted flag. Caller-side read; the database must be addressed by name.

get_doc_versions(Db, DocId)

-spec get_doc_versions(binary(), binary()) -> {ok, [map()]} | {error, term()}.

Every version of a document still resolvable: the current winner plus conflict and superseded siblings.

get_docs(Db, DocIds)

-spec get_docs(binary() | pid(), [binary()]) -> [{ok, map()} | {error, term()}] | {error, term()}.

Get multiple documents by ID (batch read).

Efficiently fetches multiple documents in a single operation using RocksDB's multi_get for improved performance over sequential reads.

Example

  Results = barrel_docdb:get_docs(<<"mydb">>, [<<"doc1">>, <<"doc2">>, <<"doc3">>]),
  %% Results = [{ok, Doc1}, {ok, Doc2}, {error, not_found}]

See also: get_docs/3.

get_docs(Db, DocIds, Opts)

-spec get_docs(binary() | pid(), [binary()], map()) -> [{ok, map()} | {error, term()}] | {error, term()}.

Get multiple documents with options (batch read).

Options

  • include_deleted - If true, include deleted documents

get_hlc()

-spec get_hlc() -> barrel_hlc:timestamp().

Get the current global HLC timestamp.

Returns the current Hybrid Logical Clock timestamp without advancing the clock. The HLC is node-global and used for ordering events across distributed machines.

Example

  TS = barrel_docdb:get_hlc().
  %% TS is a #timestamp{wall_time, logical} record

get_local_doc(Db, DocId)

-spec get_local_doc(binary() | pid(), binary()) -> {ok, map()} | {error, not_found}.

Get a local document.

get_system_doc(DocId)

-spec get_system_doc(binary()) -> {ok, map()} | {error, not_found}.

Get a global system document.

get_version_body(Db, DocId, VersionToken)

-spec get_version_body(binary(), binary(), binary()) -> {ok, map()} | {error, term()}.

The body of one version of a document (current or archived).

history_floor(Db)

-spec history_floor(binary()) -> barrel_hlc:timestamp() | undefined | {error, term()}.

The oldest HLC the history is complete from (undefined means complete since database creation).

list_attachments(Db, DocId)

-spec list_attachments(binary() | pid(), binary()) -> [binary()].

List all attachments for a document.

Returns the names of all attachments associated with a document.

Example

  AttNames = barrel_docdb:list_attachments(<<"mydb">>, <<"doc1">>).
  %% Returns [<<"file1.txt">>, <<"image.png">>]

list_branches(Parent)

-spec list_branches(binary()) -> [binary()].

The OPEN branches of a database (a branch that exists on disk but is not running is not listed).

list_dbs()

-spec list_dbs() -> [binary()].

List all open databases.

Returns the names of all currently open databases.

Example

  DbNames = barrel_docdb:list_dbs().
  %% Returns [<<"db1">>, <<"db2">>, ...]

merge_branch(Branch, Opts)

-spec merge_branch(binary(), map()) -> {ok, map()} | {error, term()}.

Merge a branch's edits back into its parent. See barrel_timeline:merge_branch/2.

new_hlc()

-spec new_hlc() -> barrel_hlc:timestamp().

Generate a new HLC timestamp.

Creates a new timestamp and advances the clock. Use this when creating events that will be sent to other nodes or need to be ordered.

Example

  TS = barrel_docdb:new_hlc().
  %% Use TS for ordering the event

node_id()

-spec node_id() -> binary().

Return this node's stable identity.

The id is generated once (hostname plus a random suffix) and persisted in the _node_id system document. It has no dependency on discovery or federation; it is a building block an external cluster or discovery layer can use to identify this node.

open_attachment_stream(Db, DocId, AttName)

-spec open_attachment_stream(binary() | pid(), binary(), binary()) -> {ok, map()} | {error, term()}.

Open a stream for reading an attachment in chunks.

For large attachments, use streaming to avoid loading the entire attachment into memory at once.

Example

  {ok, Stream} = barrel_docdb:open_attachment_stream(<<"mydb">>, <<"doc1">>, <<"large.bin">>),
  stream_to_file(Stream, File).
 
  stream_to_file(Stream, File) ->
      case barrel_docdb:read_attachment_chunk(Stream) of
          {ok, Chunk, Stream2} ->
              file:write(File, Chunk),
              stream_to_file(Stream2, File);
          eof ->
              ok
      end.

open_attachment_writer(Db, DocId, AttName, ContentType)

-spec open_attachment_writer(binary() | pid(), binary(), binary(), binary()) ->
                                {ok, map()} | {error, term()}.

Open a stream for writing an attachment in chunks.

For large attachments, use streaming to avoid loading the entire attachment into memory at once.

Example

  {ok, Writer} = barrel_docdb:open_attachment_writer(<<"mydb">>, <<"doc1">>,
                                                     <<"large.bin">>, <<"application/octet-stream">>),
  {ok, Writer2} = barrel_docdb:write_attachment_chunk(Writer, Chunk1),
  {ok, Writer3} = barrel_docdb:write_attachment_chunk(Writer2, Chunk2),
  {ok, AttInfo} = barrel_docdb:finish_attachment_writer(Writer3).

open_attachment_writer(Db, DocId, AttName, ContentType, Opts)

-spec open_attachment_writer(binary() | pid(), binary(), binary(), binary(), map()) ->
                                {ok, map()} | {error, term()}.

Open an attachment writer with options (origin_hlc, expected_digest: both checked at finish before anything commits).

open_db(Name)

-spec open_db(binary()) -> {ok, pid()} | {error, term()}.

Open an existing database.

Returns the pid of an already running database. Databases are automatically opened when created and remain open until explicitly closed or the application stops.

Example

  {ok, Pid} = barrel_docdb:open_db(<<"mydb">>).

outbox_ack(Db, Tag, Hlcs)

-spec outbox_ack(binary() | pid(), binary(), [barrel_hlc:timestamp()]) -> ok | {error, term()}.

Acknowledge processed outbox entries by their exact HLC keys.

Ack-by-exact-key is race-free: if a document was rewritten while its entry was being processed, the new entry lives at a different HLC key and re-drives the consumer; deleting the processed key never loses work.

outbox_fold(Db, Tag, Fun, Acc)

-spec outbox_fold(binary(),
                  binary(),
                  fun((barrel_outbox:entry(), term()) -> {ok, term()} | {stop, term()}),
                  term()) ->
                     term() | {error, term()}.

Fold over pending outbox entries for a tag, in HLC order.

Entries are written atomically with the document batch when a write carries the outbox => [Tag] option (see put_doc/3). The fold runs in the caller's process against the store; the database must be addressed by name and open. Fun(Entry, Acc) receives #{hlc, id, rev, deleted} and returns {ok, Acc} or {stop, Acc}.

See also: outbox_ack/3.

outbox_fold(Db, Tag, Fun, Acc, Opts)

-spec outbox_fold(binary(),
                  binary(),
                  fun((barrel_outbox:entry(), term()) -> {ok, term()} | {stop, term()}),
                  term(),
                  map()) ->
                     term() | {error, term()}.

Fold over pending outbox entries with options (limit).

put_attachment(Db, DocId, AttName, Data)

-spec put_attachment(binary() | pid(), binary(), binary(), binary()) -> {ok, map()} | {error, term()}.

Attach binary data to a document.

Stores a binary attachment associated with a document. Attachments are stored in a separate BlobDB-enabled RocksDB instance optimized for large binary data.

Example

  Data = <<"Hello, World!">>,
  {ok, Info} = barrel_docdb:put_attachment(<<"mydb">>, <<"doc1">>,
      <<"greeting.txt">>, Data).

put_attachment(Db, DocId, AttName, Data, Opts)

-spec put_attachment(binary() | pid(), binary(), binary(), binary(), map()) ->
                        {ok, map()} | {ok, ignored} | {error, term()}.

Store an attachment with options: sync, content_type, origin_hlc (replicated writes; the last-write-wins guard may answer {ok, ignored}) and expected_digest (verified before commit).

put_doc(Db, Doc)

-spec put_doc(binary() | pid(), map()) -> {ok, map()} | {error, term()}.

Create or update a document.

Stores a document in the database. If the document has an <<"id">> key, that ID is used; otherwise, a unique ID is generated.

For updates, the document must include the current <<"_rev">> value. This ensures optimistic concurrency control.

Example

  %% Create a new document
  {ok, Result} = barrel_docdb:put_doc(<<"mydb">>, #{
      <<"type">> => <<"user">>,
      <<"name">> => <<"Alice">>
  }),
  DocId = maps:get(<<"id">>, Result),
  Rev = maps:get(<<"rev">>, Result).

See also: put_doc/3.

put_doc(Db, Doc, Opts)

-spec put_doc(binary() | pid(), map(), map()) -> {ok, map()} | {error, term()}.

Create or update a document with options.

Same as put_doc/2 but accepts additional options.

Options

  • replicate => sync - Wait for document to reach replicas before returning
  • wait_for => [Target] - List of targets to wait for (requires replicate => sync)

Example

  %% Sync write - wait for document to reach nodeC
  {ok, _} = barrel_docdb:put_doc(<<"mydb">>, Doc, #{
      replicate => sync,
      wait_for => [<<"http://nodeC:8080/db/mydb">>]
  }).

put_docs(Db, Docs)

-spec put_docs(binary() | pid(), [map()]) -> [{ok, map()} | {error, term()}].

Put multiple documents in a single batch.

Efficiently writes multiple documents in a single RocksDB batch operation. This is significantly faster than calling put_doc multiple times when inserting many documents.

Example

  Docs = [
      #{<<"id">> => <<"doc1">>, <<"name">> => <<"Alice">>},
      #{<<"id">> => <<"doc2">>, <<"name">> => <<"Bob">>},
      #{<<"id">> => <<"doc3">>, <<"name">> => <<"Charlie">>}
  ],
  Results = barrel_docdb:put_docs(<<"mydb">>, Docs),
  %% Results = [{ok, #{...}}, {ok, #{...}}, {ok, #{...}}]

See also: put_docs/3.

put_docs(Db, Docs, Opts)

-spec put_docs(binary() | pid(), [map()], map()) -> [{ok, map()} | {error, term()}].

Put multiple documents with options.

Options

  • sync - If true, sync to disk before returning (default: false)

put_local_doc(Db, DocId, Doc)

-spec put_local_doc(binary() | pid(), binary(), map()) -> ok | {error, term()}.

Store a local document.

Local documents are stored in the database but are NOT replicated. They are typically used for storing replication checkpoints and other metadata.

Example

  ok = barrel_docdb:put_local_doc(<<"mydb">>, <<"_local/checkpoint">>, #{
      <<"last_seq">> => <<"100">>
  }).

put_system_doc(DocId, Doc)

-spec put_system_doc(binary(), map()) -> ok | {error, term()}.

Store a global system document.

System documents are stored in the _barrel_system database and are NOT replicated. They are used for global configuration and state.

Example

  ok = barrel_docdb:put_system_doc(<<"global_config">>, #{
      <<"max_dbs">> => 100
  }).

put_version(Db, Doc, VersionToken, VVBin, Deleted)

-spec put_version(binary() | pid(), map(), binary(), binary(), boolean()) ->
                     {ok, binary(), binary()} | {error, term()}.

Apply a replicated version (the version-vector protocol write).

The version token and vector come from the source database and are preserved; only the change-sequence HLC is issued locally. Outcomes: already-covered versions are idempotent no-ops, dominating versions fast-forward the document, concurrent versions create a conflict sibling with a deterministic last-write-wins winner.

query(Db, Bql)

-spec query(binary() | pid(), binary() | string()) -> {ok, [map()], map()} | {error, term()}.

Run a BQL query (see barrel_bql) against document indexes.

Table-function sources and SUBSCRIBE need the barrel module: this entry point is documents only.

Example

  {ok, Rows, Meta} = barrel_docdb:query(<<"mydb">>,
      <<"SELECT title FROM db WHERE type = 'post' LIMIT 10">>),
  {ok, Rows2, _} = barrel_docdb:query(<<"mydb">>,
      <<"SELECT * FROM db WHERE org = $org">>,
      #{params => #{<<"org">> => <<"acme">>}}).

Streamable queries (no ORDER BY, no UNNEST) keep find/3's chunk semantics: Meta carries has_more and a continuation to pass back in Opts. ORDER BY and UNNEST materialize the full result.

query(Db, Bql, Opts)

-spec query(binary() | pid(), binary() | string(), map()) -> {ok, [map()], map()} | {error, term()}.

read_attachment_chunk(Stream)

-spec read_attachment_chunk(map()) -> {ok, binary(), map()} | eof | {error, term()}.

Read the next chunk from an attachment stream.

rebuild_attachment_feed(Db)

-spec rebuild_attachment_feed(binary() | pid()) -> {ok, map()} | {error, term()}.

Maintenance: synthesize feed rows for attachments written before the feed existed (they do not sync otherwise). Rebuilt rows carry the minimum origin, so any real write wins over them.

resolve_conflict(Db, DocId, BaseRev, Resolution)

-spec resolve_conflict(binary() | pid(), binary(), binary(), {choose, binary()} | {merge, map()}) ->
                          {ok, map()} | {error, term()}.

Resolve a document conflict.

Allows explicit resolution of conflicting revisions. Two resolution strategies are supported:

Choose Resolution

Pick one of the existing revisions as the winner. All other branches are marked as deleted.

  {ok, Result} = barrel_docdb:resolve_conflict(<<"mydb">>, <<"doc1">>,
      <<"2-winner">>, {choose, <<"2-abc">>}).

Merge Resolution

Provide a new merged document that supersedes all conflicting branches.

  MergedDoc = #{<<"name">> => <<"Merged Name">>, <<"value">> => 42},
  {ok, Result} = barrel_docdb:resolve_conflict(<<"mydb">>, <<"doc1">>,
      <<"2-winner">>, {merge, MergedDoc}).

set_doc_embedding(Db, DocId, ExpectedRev, Vector)

-spec set_doc_embedding(binary() | pid(), binary(), binary(), [number()]) ->
                           ok | {error, conflict | not_found | term()}.

Store a computed embedding as a document's embedding column.

Embeddings are derived data kept outside the body: this never bumps the revision or emits a change. CAS on ExpectedRev: a conflict means a newer write exists. Read back with the include_embedding option of get_doc/3.

subscribe(DbName, Pattern)

-spec subscribe(db_name(), binary()) -> {ok, reference()} | {error, term()}.

Subscribe to document changes matching a path pattern.

Subscribes the calling process to receive notifications for document changes that match the given MQTT-style path pattern. Notifications are sent as messages of the form:

{barrel_change, DbName, #{id => DocId, rev => Rev, hlc => Hlc, deleted => boolean(), paths => [binary()]}}

Pattern Syntax

Patterns use MQTT-style wildcards:

  • + matches exactly one path level (e.g., <<"users/+/profile">>)
  • # matches zero or more levels (e.g., <<"orders/#">>)

Paths are derived from document field structure: #{<<"users">> => #{<<"123">> => #{<<"name">> => <<"Alice">>}}} produces paths like <<"users/123/name/Alice">>.

Example

  %% Subscribe to all user profile changes
  {ok, SubRef} = barrel_docdb:subscribe(<<"mydb">>, <<"users/+/profile/#">>),
 
  %% Receive notifications
  receive
      {barrel_change, <<"mydb">>, Change} ->
          io:format("Document ~s changed~n", [maps:get(id, Change)])
  end,
 
  %% Unsubscribe when done
  ok = barrel_docdb:unsubscribe(SubRef).

See also: unsubscribe/1.

subscribe(DbName, Pattern, Opts)

-spec subscribe(db_name(), binary(), map()) -> {ok, reference()} | {error, term()}.

Subscribe to document changes with options.

Same as subscribe/2 but with additional options.

Options

Currently no options are supported (reserved for future use).

See also: subscribe/2, unsubscribe/1.

subscribe_changes(Db, Since)

-spec subscribe_changes(binary() | pid(), barrel_hlc:timestamp() | first) ->
                           {ok, pid()} | {error, term()}.

Subscribe to a changes stream.

Returns a stream pid that can be used to iterate over changes as they occur.

Example

  {ok, Stream} = barrel_docdb:subscribe_changes(<<"mydb">>, first),
  %% Use barrel_changes_stream:next/1 to get changes

See also: subscribe_changes/3.

subscribe_changes(Db, Since, Opts)

-spec subscribe_changes(binary() | pid(), barrel_hlc:timestamp() | first, map()) ->
                           {ok, pid()} | {error, term()}.

Subscribe to a changes stream with options.

subscribe_query(DbName, Query)

-spec subscribe_query(db_name(), barrel_query:query_spec()) -> {ok, reference()} | {error, term()}.

Subscribe to document changes matching a query.

Subscribes to document change notifications for documents that match the specified query. Only documents that match the query conditions will trigger notifications.

Query subscriptions are optimized using path extraction - the full query is only evaluated when a change affects paths referenced by the query.

Example

  %% Subscribe to changes for user documents
  Query = #{where => [{path, [<<"type">>], <<"user">>}]},
  {ok, SubRef} = barrel_docdb:subscribe_query(<<"mydb">>, Query),
 
  %% Receive notifications
  receive
      {barrel_query_change, <<"mydb">>, #{id := DocId, rev := Rev}} ->
          io:format("Document ~s changed~n", [DocId])
  end,
 
  %% Unsubscribe when done
  ok = barrel_docdb:unsubscribe_query(SubRef).

See also: subscribe_query/3, unsubscribe_query/1.

subscribe_query(DbName, Query, Opts)

-spec subscribe_query(db_name(), barrel_query:query_spec(), map()) ->
                         {ok, reference()} | {error, term()}.

Subscribe to document changes matching a query with options.

Same as subscribe_query/2 but with additional options.

See also: subscribe_query/2, unsubscribe_query/1.

sweep_retention(Db)

-spec sweep_retention(binary() | pid()) -> {ok, map()} | {error, term()}.

Run a retention sweep now (the database also sweeps on a timer). Returns sweep statistics and the new floor.

sweep_ttl(Db)

-spec sweep_ttl(binary() | pid()) -> {ok, non_neg_integer()} | {error, term()}.

Run one doc TTL sweep pass now (see the ttl_sweep_interval database option). Returns the number of docs tombstoned.

sync_hlc(RemoteHlc)

-spec sync_hlc(barrel_hlc:timestamp()) -> {ok, barrel_hlc:timestamp()} | {error, clock_skew}.

Synchronize with a remote HLC timestamp.

Call this when receiving data from another node to maintain causality. The local clock is updated to reflect the remote timestamp, ensuring that subsequent events are ordered after the received data.

Example

  %% When receiving data from another node:
  RemoteHlc = ... %% HLC from remote node
  {ok, NewHlc} = barrel_docdb:sync_hlc(RemoteHlc).

unsubscribe(SubRef)

-spec unsubscribe(reference()) -> ok.

Unsubscribe from document change notifications.

Removes a subscription previously created with subscribe/2 or subscribe/3. After calling this function, no more notifications will be received for the given subscription.

Example

  {ok, SubRef} = barrel_docdb:subscribe(<<"mydb">>, <<"users/#">>),
  %% ... receive some notifications ...
  ok = barrel_docdb:unsubscribe(SubRef).

See also: subscribe/2.

unsubscribe_query(SubRef)

-spec unsubscribe_query(reference()) -> ok.

Unsubscribe from query-based change notifications.

Removes a query subscription previously created with subscribe_query/2 or subscribe_query/3.

Example

  {ok, SubRef} = barrel_docdb:subscribe_query(<<"mydb">>, Query),
  %% ... receive some notifications ...
  ok = barrel_docdb:unsubscribe_query(SubRef).

See also: subscribe_query/2.

validate_db_name(Name)

-spec validate_db_name(binary()) -> ok | {error, invalid_db_name}.

Validate a database name. Accepts: lowercase alphanumerics, underscore, hyphen. First char must be alphanumeric. Length 1..63. Internal system databases (prefix _) are accepted to support _barrel_system and similar.

write_attachment_chunk(Writer, Data)

-spec write_attachment_chunk(map(), binary()) -> {ok, map()} | {error, term()}.

Write a chunk of data to an attachment writer.