Replication Guide

barrel_docdb replicates documents between databases using a version-vector protocol: each write is a version (<hex(hlc)>@<author>), each document carries a version vector, and the target decides what it is missing by vector containment. This guide covers:

• Basic replication between local databases
• Continuous replication tasks
• Filtered replication (by path pattern or query)
• How the version-vector protocol works
• HLC-based checkpoints and incremental sync
• Transports (local and HTTP)

Basic Replication

One-Shot Replication

The simplest form of replication copies all documents from source to target:

%% Create source and target databases
{ok, _} = barrel_docdb:create_db(<<"source">>),
{ok, _} = barrel_docdb:create_db(<<"target">>),

%% Add documents to source
{ok, _} = barrel_docdb:put_doc(<<"source">>, #{
    <<"id">> => <<"doc1">>,
    <<"value">> => <<"hello">>
}),
{ok, _} = barrel_docdb:put_doc(<<"source">>, #{
    <<"id">> => <<"doc2">>,
    <<"value">> => <<"world">>
}),

%% Replicate source -> target
{ok, Result} = barrel_rep:replicate(<<"source">>, <<"target">>),

%% Check results
DocsWritten = maps:get(docs_written, Result),
io:format("Replicated ~p documents~n", [DocsWritten]).

Replication with Options

Control batch size and checkpoint frequency:

{ok, Result} = barrel_rep:replicate(<<"source">>, <<"target">>, #{
    batch_size => 50,       %% Process 50 changes per batch
    checkpoint_size => 25   %% Write checkpoint every 25 documents
}).

Result Statistics

The replication result includes statistics:

#{
    ok := true,                 %% Replication succeeded
    docs_read := 10,            %% Documents read from source
    docs_written := 10,         %% Documents written to target
    doc_read_failures := 0,     %% Failed reads
    doc_write_failures := 0,    %% Failed writes
    start_seq := first,         %% Starting HLC timestamp
    last_seq := #timestamp{},   %% Final HLC timestamp
    att_sync := skipped         %% Attachment phase outcome
} = Result.

Continuous Replication

barrel_rep:replicate/2,3 runs once and returns. For replication that keeps running and syncs changes as they happen (and survives node restarts), start a task with barrel_rep_tasks:

{ok, TaskId} = barrel_rep_tasks:start_task(#{
    source => <<"mydb">>,
    target => <<"http://remote:8080/db/mydb">>,
    mode => continuous,
    direction => push
}).

%% Manage the task
ok = barrel_rep_tasks:pause_task(TaskId),
ok = barrel_rep_tasks:resume_task(TaskId),
Tasks = barrel_rep_tasks:list_tasks().

Tasks move through pending, running, paused, completed (one-shot), and failed. Task config is persisted in a system database, so a task resumes from its checkpoint after a restart.

Attachments

Tasks replicate attachments by default, same as barrel_rep:replicate/2,3. Attachments live on their own feed (independent of the document changes feed), so they have their own checkpoint (att_seq, persisted alongside last_seq) and their own idle-wake handling: a continuous task periodically re-checks attachments even when no document change occurs, so an attachment added to an already-replicated document still converges without any further doc write.

{ok, TaskId} = barrel_rep_tasks:start_task(#{
    source => <<"mydb">>,
    target => <<"http://remote:8080/db/mydb">>,
    mode => continuous,
    direction => push,
    attachments => true,      %% default; set false to skip attachments
    att_batch_size => 100      %% attachment feed batch size (default 100)
}).

Set attachments => false to replicate documents only. This is persisted with the rest of the task config, so it survives pause_task/resume_task and node restarts.

Attachment Storage Backends

Where attachment bytes actually live is pluggable, via the barrel_att_backend behaviour: barrel_att_store picks a backend per database from att_opts.backend (default is barrel_att_store_blob, an embedded RocksDB instance with BlobDB enabled). barrel_att_s3 (a separate sibling app, opt in with the s3 rebar3 profile) is an S3-compatible backend for AWS S3, MinIO, or Garage -- see its getting started guide.

This matters for replication because the behaviour splits its callbacks in two:

• Required: put/get/delete/streaming, storing and reading bytes. Any backend needs these.
• Optional: att_changes/4, att_floor/2, sweep_att_feed/3, rebuild_feed/2 (the attachment change feed replication reads from), checkpoint/2 (timeline branching), destroy/2 (backend-owned state beyond the local directory, erased on delete_db).

barrel_att_store_blob implements both sets: it keeps its feed in the same RocksDB instance as the blobs, committed in the same write batch as the blob write, so the feed can never fall out of sync with what was actually stored. barrel_att_s3 implements both sets too, as of a later revision than its first release: it keeps its feed in a small local RocksDB instance alongside the (remote) attachment bytes, since S3 has no equivalent to a local atomic write batch spanning both -- committed just after each S3 write succeeds, not atomically with it. Either way, replication treats both backends the same: full LWW-checked sync as both source and target.

A backend that implements only the required set (put/get/delete/streaming, no feed at all) still degrades gracefully rather than erroring, detected per call via barrel_att_store:supports_sync/1 (erlang:function_exported/3): attachment sync reports att_sync => skipped as a replication source, for one-shot replication, timeline merges, and tasks alike. Attachments still work locally (put/get/delete) in that case; they just don't replicate. This check only covers the source -- such a backend as a replication target is not detected, since put/delete are required callbacks and land regardless, with no feed on the target to check origin_hlc against, so the last-write-wins guard a feed-backed target enforces silently does not apply for it.

Filtered Replication

Replicate only documents matching specific criteria using the filter option. A filtered stream keeps its own checkpoint, separate from the full replication.

Filter by Path Pattern

Use MQTT-style path patterns to select documents:

%% Replicate only user documents
{ok, Result} = barrel_rep:replicate(<<"source">>, <<"target">>, #{
    filter => #{
        paths => [<<"type/user/#">>]
    }
}).

%% Replicate orders and invoices
{ok, Result} = barrel_rep:replicate(<<"source">>, <<"target">>, #{
    filter => #{
        paths => [<<"type/order/#">>, <<"type/invoice/#">>]
    }
}).

Filter by Query

Use declarative queries for complex filtering:

%% Replicate only active users
{ok, Result} = barrel_rep:replicate(<<"source">>, <<"target">>, #{
    filter => #{
        query => #{
            where => [
                {path, [<<"type">>], <<"user">>},
                {path, [<<"active">>], true}
            ]
        }
    }
}).

Combined Filters (AND Logic)

When both paths and query are specified, documents must match both filters:

%% Replicate only active users (path AND query must match)
{ok, Result} = barrel_rep:replicate(<<"source">>, <<"target">>, #{
    filter => #{
        paths => [<<"type/user/#">>],      %% Must match path pattern
        query => #{                        %% AND must match query
            where => [{path, [<<"active">>], true}]
        }
    }
}).

Path Pattern Syntax

How Replication Works

The Replication Algorithm

Replication makes one diff round-trip per batch of changes, then reads and applies each missing document. Conflict handling lives entirely in the target's put_version, so the algorithm needs no ancestor negotiation.

1. Read checkpoint: Find the last replicated HLC timestamp.
2. Fetch changes: Get changes from source since that timestamp (optionally filtered). Each change carries the document's current version token.
3. Sync HLC: Advance the target clock past the newest source change so the target's change sequence stays strictly increasing.
4. Diff versions: Call diff_versions on the target with #{DocId => Token}. The target answers have (its version vector already covers the offered version) or missing.
5. Transfer documents: For each missing document, read it from the source with its version, vector, and deleted flag.
6. Apply to target: Call put_version. The target no-ops, fast-forwards, or records a conflict sibling (see below).
7. Save checkpoint: Record progress with the HLC for incremental sync.

Versions and Version Vectors

barrel_docdb uses MVCC based on Hybrid Logical Clocks, not revision trees.

• A version is {HLC, Author}. Its API token is <hex(hlc)>@<author>: the HLC gives causal last-write-wins ordering and the author is the id of the database that made the write (per-database, so two databases on one node detect each other's writes as concurrent). This is the value in the <<"_rev">> key.
• A version vector maps each author to the highest HLC this replica has seen from it for the document. Comparing two vectors decides how a replicated write relates to the local one:

%% Create document
{ok, #{<<"rev">> := V1}} = barrel_docdb:put_doc(Db, Doc),
%% V1 = <<"0000018abc...@f1e0...">>

%% Update produces a new version with a later HLC
{ok, #{<<"rev">> := V2}} = barrel_docdb:put_doc(Db, UpdatedDoc#{<<"_rev">> => V1}).

diff_versions - Finding Missing Versions

diff_versions answers, per document, whether the target already covers an offered version:

{ok, Diff} = barrel_docdb:diff_versions(<<"target">>, #{
    <<"doc1">> => V1,
    <<"doc2">> => V2
}),
%% Diff = #{<<"doc1">> => have, <<"doc2">> => missing}
%% `have`    = the doc's version vector contains the offered version
%% `missing` = the target needs it

put_version - Applying a Version

put_version applies a version read from the source. The source's token and vector are preserved; only the change-sequence HLC is issued locally. The outcome depends on how the vectors relate:

• Already covered (contains): idempotent no-op.
• Remote dominates: fast-forward. The old winner is retained as a superseded version.
• Concurrent: an optional per-database conflict_merger hook may resolve it with a merged body; otherwise last-write-wins by version picks the winner and the loser stays live as a retained conflict sibling. The vectors merge either way.

{ok, Doc, #{version := Token, vv := VVBin, deleted := Del}} =
    barrel_docdb:get_doc_for_replication(<<"source">>, <<"doc1">>),
{ok, _DocId, _Winner} =
    barrel_docdb:put_version(<<"target">>, Doc, Token, VVBin, Del).

Checkpoints and Incremental Replication

How Checkpoints Work

Checkpoints are stored as local documents (not replicated) in both databases:

%% Checkpoints are stored at:
%% _local/replication-checkpoint-<rep_id>

%% They contain the last replicated HLC and session history:
#{
    <<"history">> => [
        #{
            <<"source_last_hlc">> => <<...>>,  %% Encoded HLC timestamp
            <<"session_id">> => <<"abc123">>,
            <<"end_time">> => <<"2024-01-15T10:30:00Z">>
        }
    ]
}

The replication id (and so the checkpoint) is derived from the source and target identities plus any filter. A filtered stream and an unfiltered stream do not share a checkpoint.

HLC-Based Ordering

Replication uses HLC (Hybrid Logical Clock) timestamps as the change sequence:

• Distributed ordering: HLC provides consistent ordering across nodes.
• Causal consistency: Events are ordered by cause-effect relationships.
• Clock synchronization: The target clock is advanced with the source's during replication.

If a checkpoint predates the source's history floor (its retention window), the run forces a full resync so current state still converges.

Incremental Sync

Subsequent replications automatically resume from the last checkpoint:

%% First replication - syncs all documents
{ok, R1} = barrel_rep:replicate(<<"source">>, <<"target">>),

%% Add more documents to source
{ok, _} = barrel_docdb:put_doc(<<"source">>, #{<<"id">> => <<"doc3">>}),

%% Second replication - only syncs new documents
{ok, R2} = barrel_rep:replicate(<<"source">>, <<"target">>).
%% R2 shows fewer docs_read than R1

Transports

The Transport Behaviour

Replication talks to databases through the barrel_rep_transport behaviour, so the same algorithm works locally or over the network:

-callback get_doc(Endpoint, DocId, Opts) ->
    {ok, Doc, Meta} | {error, term()}.
    %% Meta = #{version := Token, vv := EncodedVV, deleted := boolean()}
-callback put_version(Endpoint, Doc, VersionToken, VVBin, Deleted) ->
    {ok, DocId, WinnerToken} | {error, term()}.
-callback diff_versions(Endpoint, TokenMap) ->
    {ok, #{DocId => missing | have}} | {error, term()}.
-callback get_changes(Endpoint, Since, Opts) -> {ok, Changes, LastSeq}.
-callback get_local_doc(Endpoint, DocId) -> {ok, Doc} | {error, not_found}.
-callback put_local_doc(Endpoint, DocId, Doc) -> ok.
-callback delete_local_doc(Endpoint, DocId) -> ok.
-callback db_info(Endpoint) -> {ok, Info}.
-callback sync_hlc(Endpoint, Hlc) -> {ok, Hlc}.

Attachment sync, plus rep_id_term/1 for checkpoint-stable endpoint identity, are optional callbacks.

Built-in Transports

barrel_rep_transport_local - for databases in the same Erlang VM (the default for both endpoints):

%% No configuration needed
{ok, _} = barrel_rep:replicate(<<"source">>, <<"target">>).

barrel_rep_transport_http - for a remote database, speaking the /db/:db/_sync/* wire served by barrel_server:

{ok, Result} = barrel_rep:replicate(
    <<"mydb">>,
    <<"http://remote:8080/db/mydb">>,
    #{
        source_transport => barrel_rep_transport_local,
        target_transport => barrel_rep_transport_http
    }).

The HTTP endpoint is a normalized URL; credentials come from auth => #{token => Bin} on the endpoint or the {barrel_docdb, sync_auth, #{Origin => Token}} app env, so secrets never land in persisted task configs.

Implementing a Custom Transport

Implement the barrel_rep_transport callbacks for another wire (TCP, a message bus, ...). Use barrel_rep_transport_http as the reference implementation.

-module(my_transport).
-behaviour(barrel_rep_transport).
-export([get_doc/3, put_version/5, diff_versions/2, get_changes/3,
         get_local_doc/2, put_local_doc/3, delete_local_doc/2,
         db_info/1, sync_hlc/2]).

%% ... implement each callback against your endpoint term

Using a custom transport:

Config = #{
    source => <<"local_source">>,
    target => #{host => "remote.example.com", db => <<"mydb">>},
    source_transport => barrel_rep_transport_local,
    target_transport => my_transport
},
{ok, Result} = barrel_rep:replicate_one_shot(Config, #{}).

Conflict Handling

How Conflicts Occur

A conflict happens when the same document is written concurrently in two databases (neither version's vector dominates the other) before replication:

Source:  doc1: v1 -> v2a
Target:  doc1: v1 -> v2b   (concurrent with v2a)

How Conflicts Are Resolved

Replication never fails on a conflict. put_version picks a deterministic last-write-wins winner (max by version: HLC first, author id as tie-break) and retains the loser as a live conflict sibling. Every replica that sees the same version set picks the same winner. A per-database conflict_merger hook, if configured, gets the first say and can supersede both sides with a merged body.

Surfacing and Resolving Conflicts

%% List the live sibling tokens that conflict with the winner
{ok, Conflicts} = barrel_docdb:get_conflicts(Db, DocId),

%% Or read them inline
{ok, Doc} = barrel_docdb:get_doc(Db, DocId, #{conflicts => true}),
Siblings = maps:get(<<"_conflicts">>, Doc, []).

Resolve by choosing an existing sibling or writing a merged body. BaseRev is the current winning token (optimistic lock):

%% Keep one version, drop the others
{ok, _} = barrel_docdb:resolve_conflict(Db, DocId, Winner, {choose, Sibling}),

%% Or write a merged body that supersedes all siblings
{ok, _} = barrel_docdb:resolve_conflict(Db, DocId, Winner, {merge, MergedDoc}).

A resolution is an ordinary write, so it replicates like any other.

Best Practices

1. Use meaningful document IDs - Makes debugging easier.
2. Handle failures gracefully - Check doc_read_failures and doc_write_failures.
3. Monitor checkpoint progress - Detect stalled replications.
4. Consider batch sizes - Larger batches are faster but use more memory.
5. Resolve conflicts - Do not ignore retained siblings; converge them with resolve_conflict.

API Reference

See the API reference for complete function documentation.

• barrel_rep - Replication public API (replicate/2,3, replicate_one_shot/1,2)
• barrel_rep_tasks - Continuous / persistent replication tasks
• barrel_rep_transport - Transport behaviour
• barrel_docdb:put_version/5 - Apply a replicated version
• barrel_docdb:diff_versions/2 - Version diff by vector containment
• barrel_docdb:get_doc_for_replication/2 - Read a document with its version metadata