macula_rag (macula_rag v0.3.0)
View SourceFederated retrieval over the Macula mesh: ask every shard of an org, merge the answers.
A SHARD is one node holding part of an index. It answers queries through register_responder/1, which provides the org's procedure <Org>/rag.query_shard_v1, and says what it holds through advertise/2. One shard per node: the node id is the shard's identity, and the shard_id given to configure/3 is a label for people.
A QUERY asks every shard of the org, in parallel, and merges their hits by score. "Every shard of the org" is macula's own list of the procedure's providers, which it has checked against the org's D25 delegations, so a node the org did not delegate is never asked. Only shards built with the same EMBEDDING (model and dimension) as this node are asked, because scores from two embeddings do not compare. Nothing is dropped: the result names every shard that answered and every one that did not, with why.
A HIT is the consumer's map. Its minimum is a binary id and a numeric score; everything else is carried as it is. A merged hit comes back with binary keys, the ones every received payload has, plus node_id and shard_id naming the shard it came from.
The shard's bloom is carried in its summary and listed by shards/0; it is not used to choose shards in this version.
Start by configure/3 once the pool is connected.
Summary
Functions
Publish what this shard holds: its topics and a bloom filter over them, with the shard id and embedding from configure/3. Republished every summary_republish_ms until withdraw/0, so a node that starts later still hears it. A shard with no summary is not asked.
The configuration in force.
Bind the library to a connected macula Pool and its Realm (the 32-byte realm id). Opts
The corpus hash of a provider's description.
Drop the configuration. Queries and advertising then answer {error, not_configured}.
Ask every shard of the org built with this node's embedding. Opts: top_k (10), the most hits returned after the merge, and timeout_ms (query_timeout_ms), how long the slowest shard is waited for.
Answer the org's queries with Responder, called with the consumer's query (binary keys, as received) and #{top_k => K}. It returns {ok, Hits}, each hit with at least a binary id and a numeric score, or {error, Reason}, which the asking node sees as this shard's failure.
The shards this node has heard a summary from, this one included.
A provider's signature over its corpus hash, with its identity key.
What this node is doing for the org
Check a description from the provider the caller pinned (macula:call/6 with provider), under the caller's profile.
ok when a hit's text is the text its provenance hashes.
Types
-type options() :: #{org := binary(), shard_id := binary(), realm_name := binary(), embedding := macula_rag_contract:embedding(), query_timeout_ms => pos_integer(), grant_retry_ms => pos_integer(), summary_republish_ms => pos_integer(), io => map()}.
-type query_report() :: macula_rag_query:report().
-type responder() :: fun((map(), #{top_k := pos_integer()}) -> {ok, [map()]} | {error, term()}).
Functions
Publish what this shard holds: its topics and a bloom filter over them, with the shard id and embedding from configure/3. Republished every summary_republish_ms until withdraw/0, so a node that starts later still hears it. A shard with no summary is not asked.
-spec configuration() -> {ok, map()} | {error, not_configured}.
The configuration in force.
Bind the library to a connected macula Pool and its Realm (the 32-byte realm id). Opts:
org the org whose shards these are; its procedure is <org>/rag.query_shard_v1 shard_id a label for this node's shard; the node id is its identity realm_name the realm's name, e.g. io.macula; its SHA-256 must be Realm embedding #{model => Binary, dim => PosInteger} this node's index uses
and, optionally, query_timeout_ms (1500), grant_retry_ms (30000), and summary_republish_ms (60000). io replaces the macula calls, for tests.
Configuring again rebinds: the summary subscription moves to the new pool, and a registered responder asks for its grant again.
The corpus hash of a provider's description.
-spec forget_configuration() -> ok.
Drop the configuration. Queries and advertising then answer {error, not_configured}.
-spec query(map()) -> {ok, [hit()], query_report()} | {error, term()}.
-spec query(map(), map()) -> {ok, [hit()], query_report()} | {error, term()}.
Ask every shard of the org built with this node's embedding. Opts: top_k (10), the most hits returned after the merge, and timeout_ms (query_timeout_ms), how long the slowest shard is waited for.
{ok, Hits, #{answered => [NodeId], failed => [{NodeId, Reason}]}}, node ids in lowercase hex. A shard is failed with no_summary or {embedding_mismatch, Theirs} without being asked, or with timeout, malformed_answer, or the error it answered. {error, {no_shards, Reason}} when macula lists no provider at all.
-spec register_responder(responder()) -> ok | {error, not_configured}.
Answer the org's queries with Responder, called with the consumer's query (binary keys, as received) and #{top_k => K}. It returns {ok, Hits}, each hit with at least a binary id and a numeric score, or {error, Reason}, which the asking node sees as this shard's failure.
Registering advertises the procedure as soon as macula grants it: that needs the org's D25 delegation naming this node. Without one, status/0 reports {not_granted, ...} and the grant is asked for again every grant_retry_ms, so a delegation made later is picked up without a restart. A link that drops and returns needs nothing: macula replays the advertisement on it.
-spec shards() -> [map()].
The shards this node has heard a summary from, this one included.
-spec sign_corpus(binary(), macula_node_keys:node_key()) -> binary().
A provider's signature over its corpus hash, with its identity key.
-spec status() -> #{responder := term(), advertised := boolean(), shards := non_neg_integer(), configured := boolean()}.
What this node is doing for the org:
responder not_registered, granted, or {not_granted, #{reason, since_ms}}: macula's reason for refusing the grant, as mcl_om's provider_grants reports it. {provider_authorization, {procedure_delegation, not_found}} means an operator has to delegate this node for the org. advertised whether this shard's summary is being published shards how many shards this node has a summary from
-spec unregister_responder() -> ok.
-spec verify_corpus(map(), <<_:256>>, macula_crypto_profile:profile()) -> macula_rag_corpus:verdict().
Check a description from the provider the caller pinned (macula:call/6 with provider), under the caller's profile.
-spec verify_hit(map()) -> ok | {error, malformed_provenance | content_mismatch}.
ok when a hit's text is the text its provenance hashes.
-spec withdraw() -> ok | {error, term()}.