Observability

Barrel DocDB provides observability through three pillars using the instrument library:

• Tracing - Distributed traces with OpenTelemetry-compatible spans
• Metrics - Prometheus-compatible metrics for monitoring
• Logging - Structured logging with trace context correlation

All three signals are automatically correlated via trace context, enabling end-to-end debugging across distributed operations.

HTTP surface

barrel_docdb has no built-in HTTP server. Metrics are collected in-process and can be rendered as Prometheus text with barrel_metrics:export_text/0. To scrape them over HTTP, or to trace incoming HTTP requests, run the barrel_server app (see the umbrella REST server guide docs/guides/rest-server.md). OTLP endpoints below refer to an OpenTelemetry collector, not to barrel_docdb.

Quick Start

Enable Console Tracing (Development)

%% In sys.config
{barrel_docdb, [
    {tracing, [
        {enabled, true},
        {exporter, console}
    ]}
]}

Enable OTLP Export for All Signals (Production)

%% In sys.config
{barrel_docdb, [
    %% Tracing - spans to OTLP
    {tracing, [
        {enabled, true},
        {exporter, otlp},
        {otlp_endpoint, "http://localhost:4318"}
    ]},
    %% Metrics - push to OTLP (in addition to in-process export)
    {metrics, [
        {exporter, otlp},
        {otlp_endpoint, "http://localhost:4318"},
        {export_interval, 60000}  %% ms, default 60s
    ]},
    %% Logging - push to OTLP
    {logging, [
        {exporter, otlp},
        {otlp_endpoint, "http://localhost:4318"}
    ]}
]}

Render Metrics In-Process

Text = barrel_metrics:export_text().

Run barrel_server to expose this over HTTP for a Prometheus scrape.

Tracing

Barrel DocDB implements OpenTelemetry-compatible distributed tracing, enabling you to trace document operations, queries, replication, and cross-node communication. When barrel_docdb runs behind barrel_server, HTTP request spans are added by that layer.

Configuration

Add tracing configuration to your sys.config:

{barrel_docdb, [
    {tracing, [
        {enabled, true},           %% Enable/disable tracing (default: true)
        {exporter, console},       %% Exporter type (see below)
        {console_format, text},    %% For console: text | json
        {otlp_endpoint, "http://localhost:4318"}  %% For OTLP
    ]}
]}

Exporters

No Export (Default)

Tracing is enabled internally but spans are not exported. Useful when you only need trace context correlation in logs.

{tracing, [{enabled, true}, {exporter, none}]}

Console Exporter

Prints spans to stdout. Useful for development and debugging.

{tracing, [
    {enabled, true},
    {exporter, console},
    {console_format, text}  %% text | json
]}

Example output (text format):

[span] get mydb (0.4ms)
  trace_id: 0af7651916cd43dd8448eb211c80319c
  span_id: b7ad6b7169203331
  attributes:
    db.system.name: barrel
    db.namespace: mydb
    db.operation.name: get
    db.document.id: doc123

OTLP Exporter

Sends spans to an OpenTelemetry Collector or compatible backend (Jaeger, Zipkin, Datadog, etc.).

{tracing, [
    {enabled, true},
    {exporter, otlp},
    {otlp_endpoint, "http://localhost:4318"}
]}

Docker Compose example with Jaeger:

Tracing is configured in sys.config. Mount a custom config into the container and point barrel at it.

services:
  barrel:
    image: barrel/barrel_docdb
    volumes:
      - ./sys.config:/app/releases/0.6.2/sys.config:ro

  jaeger:
    image: jaegertracing/all-in-one:latest
    ports:
      - "16686:16686"  # Jaeger UI
      - "4318:4318"    # OTLP HTTP

With sys.config:

[{barrel_docdb, [
    {tracing, [
        {enabled, true},
        {exporter, otlp},
        {otlp_endpoint, "http://jaeger:4318"}
    ]}
]}].

Custom Exporter

Use any module implementing the instrument_exporter behaviour:

{tracing, [
    {enabled, true},
    {exporter, {my_custom_exporter, #{option1 => value1}}}
]}

Trace Context Propagation

Barrel DocDB automatically propagates trace context using W3C Trace Context headers (traceparent, tracestate).

Outgoing requests: During replication, trace context is injected into HTTP headers for cross-node correlation.

Incoming HTTP requests: When barrel_docdb runs behind barrel_server, that layer extracts trace context from request headers and adds the traceparent header to responses.

Instrumented Operations

Semantic Conventions

Barrel DocDB follows OpenTelemetry Database Semantic Conventions:

Metrics

Barrel DocDB supports two modes for metrics export:

• Pull model - barrel_server exposes the in-process metrics for Prometheus to scrape
• Push model - Metrics are pushed to an OTLP collector

Configuration

{barrel_docdb, [
    {metrics, [
        {exporter, none},              %% none | otlp | console
        {otlp_endpoint, "http://localhost:4318"},
        {export_interval, 60000},      %% Push interval in ms (default: 60s)
        {otlp_headers, #{}},           %% Optional auth headers
        {otlp_compression, none}       %% none | gzip
    ]}
]}

Pull Model (Prometheus Scraping)

Metrics are always recorded in-process regardless of push configuration. Render them in Prometheus text format:

Text = barrel_metrics:export_text().

Run barrel_server to serve this output on its metrics endpoint for a Prometheus scrape.

Available Metrics

Document Operations

Operations: put, get, delete

Example:

barrel_doc_operations{db="mydb",operation="put"} 1234
barrel_doc_operation_duration_seconds_bucket{db="mydb",operation="get",le="0.01"} 5000

Query Operations

Replication

Storage

HTTP request metrics

Metrics for HTTP requests (method, path, status, latency) are produced by barrel_server, not barrel_docdb, since barrel_docdb has no HTTP surface of its own.

Prometheus Configuration (Pull)

Point Prometheus at the barrel_server listener that exposes the metrics:

# prometheus.yml
scrape_configs:
  - job_name: 'barrel'
    static_configs:
      - targets: ['localhost:8080']
    metrics_path: /metrics
    scrape_interval: 15s

Push Model (OTLP)

To push metrics directly to an OpenTelemetry Collector:

{barrel_docdb, [
    {metrics, [
        {exporter, otlp},
        {otlp_endpoint, "http://otel-collector:4318"}
    ]}
]}

With authentication:

{barrel_docdb, [
    {metrics, [
        {exporter, otlp},
        {otlp_endpoint, "https://otlp.vendor.com:4318"},
        {otlp_headers, #{
            <<"Authorization">> => <<"Bearer your-api-key">>
        }},
        {otlp_compression, gzip}
    ]}
]}

OpenTelemetry Collector configuration for metrics:

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"
  # Or send to a metrics backend
  otlphttp:
    endpoint: "https://metrics-backend.example.com"

service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [prometheus]

Grafana Dashboard Queries

Document Operation Rate

rate(barrel_doc_operations[5m])

Replication Throughput

increase(barrel_replication_docs[1m])

P99 Document Latency

histogram_quantile(0.99, rate(barrel_doc_operation_duration_seconds_bucket[5m]))

Documents per Second

rate(barrel_doc_operations{operation="put"}[5m])

Query Throughput

rate(barrel_query_operations[5m])

Alerting Examples

# Prometheus alerting rules
groups:
  - name: barrel_docdb
    rules:
      - alert: BarrelHighLatency
        expr: |
          histogram_quantile(0.99, rate(barrel_doc_operation_duration_seconds_bucket[5m])) > 0.1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High P99 latency on Barrel DocDB"

      - alert: BarrelReplicationErrors
        expr: increase(barrel_replication_errors[5m]) > 0
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "Replication errors detected"

Logging

Barrel DocDB uses Erlang's built-in logger module with automatic trace context enrichment. Logs can be:

• Written to stdout (default Erlang logger)
• Written to files with rotation
• Pushed to an OTLP collector for centralized logging

Configuration

{barrel_docdb, [
    {logging, [
        {exporter, none},              %% none | console | file | otlp

        %% File exporter options
        {file_path, "/var/log/barrel.log"},
        {file_format, text},           %% text | json
        {file_max_size, 10485760},     %% 10MB, 0 = unlimited
        {file_max_files, 5},           %% Number of rotated files
        {file_compress, false},        %% Compress rotated files

        %% OTLP exporter options
        {otlp_endpoint, "http://localhost:4318"},
        {otlp_headers, #{}},           %% Optional auth headers
        {otlp_compression, none}       %% none | gzip
    ]}
]}

Standard Output (Default)

Configure logging in sys.config:

{kernel, [
    {logger_level, info},
    {logger, [
        {handler, default, logger_std_h, #{
            level => info,
            formatter => {logger_formatter, #{
                template => [time, " [", level, "] ", pid, " ", msg, "\n"]
            }}
        }}
    ]}
]}

Trace Context in Logs

When tracing is enabled, log messages are automatically enriched with trace context:

%% In your code
logger:info("Processing document ~s", [DocId])

Output with trace context:

2024-01-15T10:30:45.123Z [info] <0.234.0> Processing document doc123 trace_id=0af7651916cd43dd8448eb211c80319c span_id=b7ad6b7169203331

JSON Logging for Production

For structured logging (recommended for production):

{kernel, [
    {logger_level, info},
    {logger, [
        {handler, default, logger_std_h, #{
            level => info,
            formatter => {logger_formatter, #{
                template => [msg],
                single_line => true
            }},
            config => #{
                type => standard_io
            }
        }}
    ]}
]}

Log Levels

File Exporter

Write logs to files with automatic rotation:

{barrel_docdb, [
    {logging, [
        {exporter, file},
        {file_path, "/var/log/barrel/app.log"},
        {file_format, json},           %% text | json
        {file_max_size, 52428800},     %% 50MB
        {file_max_files, 10},          %% Keep 10 rotated files
        {file_compress, true}          %% Compress rotated files (.gz)
    ]}
]}

File rotation behavior:

• When file_max_size is reached, files rotate: app.log -> app.log.1 -> app.log.2 -> ...
• If file_compress is true, rotated files become app.log.1.gz, etc.
• Set file_max_size to 0 to disable rotation

JSON format output:

{"timeUnixNano":"1705312245123456000","severityText":"INFO","body":{"stringValue":"Processing document doc123"},"traceId":"0af7651916cd43dd8448eb211c80319c","spanId":"b7ad6b7169203331"}

Text format output:

[2024-01-15T10:30:45.123456Z] INFO [trace_id=0af7651916cd43dd span_id=b7ad6b7169203331] Processing document doc123

Push Model (OTLP)

To push logs directly to an OpenTelemetry Collector:

{barrel_docdb, [
    {logging, [
        {exporter, otlp},
        {otlp_endpoint, "http://otel-collector:4318"}
    ]}
]}

With authentication:

{barrel_docdb, [
    {logging, [
        {exporter, otlp},
        {otlp_endpoint, "https://logs.vendor.com:4318"},
        {otlp_headers, #{
            <<"Authorization">> => <<"Bearer your-api-key">>
        }},
        {otlp_compression, gzip}
    ]}
]}

OpenTelemetry Collector configuration for logs:

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

exporters:
  loki:
    endpoint: "http://loki:3100/loki/api/v1/push"
  # Or send to another backend
  elasticsearch:
    endpoints: ["http://elasticsearch:9200"]

service:
  pipelines:
    logs:
      receivers: [otlp]
      exporters: [loki]

Correlating Logs with Traces

Use the trace ID from logs to find the corresponding trace in your tracing backend:

1. Find the error in logs:

   2024-01-15T10:30:45.123Z [error] Document conflict trace_id=0af7651916cd43dd8448eb211c80319c

2. Search for the trace ID in Jaeger/Zipkin/etc.

3. View the full trace to understand the request flow

Complete Example Setup

The barrel service below runs the barrel_server app, which embeds barrel_docdb and exposes both the REST API and the metrics scrape endpoint.

Docker Compose with Full Observability

version: '3.8'

services:
  barrel:
    image: barrel/barrel_server
    ports:
      - "8080:8080"
    volumes:
      - barrel-data:/data
      - ./sys.config:/etc/barrel/sys.config

  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "4318:4318"

  jaeger:
    image: jaegertracing/all-in-one:latest
    ports:
      - "16686:16686"

  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml

  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin

volumes:
  barrel-data:

Barrel Configuration (sys.config)

%% sys.config - Full OTLP export for all signals
[
    {barrel_docdb, [
        {data_dir, "/data/barrel"},

        %% Tracing - push spans to OTLP
        {tracing, [
            {enabled, true},
            {exporter, otlp},
            {otlp_endpoint, "http://otel-collector:4318"}
        ]},

        %% Metrics - push to OTLP (also rendered in-process)
        {metrics, [
            {exporter, otlp},
            {otlp_endpoint, "http://otel-collector:4318"},
            {export_interval, 30000}
        ]},

        %% Logging - push to OTLP
        {logging, [
            {exporter, otlp},
            {otlp_endpoint, "http://otel-collector:4318"}
        ]}
    ]},
    {kernel, [
        {logger_level, info}
    ]}
].

OpenTelemetry Collector Configuration

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024

exporters:
  # Traces to Jaeger
  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true

  # Metrics to Prometheus
  prometheus:
    endpoint: "0.0.0.0:8889"

  # Logs to Loki
  loki:
    endpoint: "http://loki:3100/loki/api/v1/push"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [loki]

Prometheus Configuration

# prometheus.yml
global:
  scrape_interval: 15s

scrape_configs:
  # Scrape the barrel_server listener (pull model)
  - job_name: 'barrel'
    static_configs:
      - targets: ['barrel:8080']
    metrics_path: /metrics

  # Or scrape from OTEL collector (if using push model)
  - job_name: 'otel-collector'
    static_configs:
      - targets: ['otel-collector:8889']

Programmatic Access

Recording Custom Spans

%% Create a custom span for your application logic
barrel_trace:with_db_span(my_operation, <<"mydb">>, fun() ->
    %% Your code here
    barrel_trace:set_attribute(<<"custom.attribute">>, <<"value">>),
    do_work()
end).

%% With extra attributes
barrel_trace:with_db_span(my_operation, <<"mydb">>, #{
    <<"custom.key">> => <<"value">>
}, fun() ->
    do_work()
end).

Recording Errors

barrel_trace:with_db_span(risky_operation, <<"mydb">>, fun() ->
    case do_something() of
        {ok, Result} -> Result;
        {error, Reason} ->
            barrel_trace:record_error(Reason),
            {error, Reason}
    end
end).

Adding Events

barrel_trace:with_db_span(batch_process, <<"mydb">>, fun() ->
    lists:foreach(fun(Item) ->
        process_item(Item),
        barrel_trace:add_event(<<"item.processed">>, #{
            <<"item.id">> => Item
        })
    end, Items)
end).

Recording Custom Metrics

%% Increment a counter
barrel_metrics:inc_doc_ops(<<"mydb">>, put),

%% Record a histogram observation
barrel_metrics:observe_doc_latency(<<"mydb">>, get, 15),  %% 15ms

%% Set a gauge
barrel_metrics:set_db_docs(<<"mydb">>, 1000).