barrel_embedding_policy (barrel v1.0.0)

View Source

Embedding policy for record-mode databases.

A policy declares which document fields embed, how they are joined into the embedded text, and how documents project into search metadata. It is configured per database at barrel:open/2 via the embedding option and evaluated on every write by the record indexer, and on every read by the read-through docstore adapter.

This module is pure: validation, matching, text rendering, and metadata projection only. Embedding itself (barrel_embed) and persistence of the policy are barrel's concern.

Policy map:

   #{
       fields          => [[<<"title">>], [<<"body">>, <<"text">>]],
       join            => <<"\n">>,          %% default
       mode            => async | sync,       %% default async
       dimensions      => pos_integer(),      %% optional, checked at open
       embedder        => term(),             %% barrel_embed config, opaque here
       metadata_fields => [binary()]          %% optional projection
   }

A field is a path of binary keys into the document (a bare binary is shorthand for a top-level field). Only binary values embed; missing fields and non-binary values are skipped. Without metadata_fields, metadata is the document minus id and _-prefixed keys.

fields is optional: a policy without fields never auto-embeds and relies on client-supplied vectors (the <<"_embedding">> document field or the vector put option).

Summary

Types

path/0
policy/0

Functions

matches(_, Doc)

Does the document carry at least one embeddable policy field? Documents that match nothing produce no vector (and any existing vector is removed by the indexer).

metadata(Policy, Doc)

Project a document into search metadata. With metadata_fields, keep only those top-level fields; otherwise the document minus id and _-prefixed keys.

mode(_)

The policy's write mode (async | sync).

text(_, Doc)

Render the text to embed: policy fields in order, missing and non-binary values skipped, joined with the policy separator.

validate(Map)

Validate and normalize a policy map. Bare-binary fields become single-segment paths; defaults are filled in (join, mode). Unknown keys are rejected so config typos fail loudly at open time.

Types

path/0

-type path() :: [binary()].

policy/0

-type policy() ::
          #{fields := [path()],
            join := binary(),
            mode := async | sync,
            dimensions => pos_integer(),
            embedder => term(),
            metadata_fields => [binary()]}.

Functions

matches(_, Doc)

-spec matches(policy(), map()) -> boolean().

Does the document carry at least one embeddable policy field? Documents that match nothing produce no vector (and any existing vector is removed by the indexer).

metadata(Policy, Doc)

-spec metadata(policy(), map()) -> map().

Project a document into search metadata. With metadata_fields, keep only those top-level fields; otherwise the document minus id and _-prefixed keys.

mode(_)

-spec mode(policy()) -> async | sync.

The policy's write mode (async | sync).

text(_, Doc)

-spec text(policy(), map()) -> binary().

Render the text to embed: policy fields in order, missing and non-binary values skipped, joined with the policy separator.

validate(Map)

-spec validate(map()) -> {ok, policy()} | {error, term()}.

Validate and normalize a policy map. Bare-binary fields become single-segment paths; defaults are filled in (join, mode). Unknown keys are rejected so config typos fail loudly at open time.