Artifact services

View Source

Erlang ADK stores immutable, versioned binary artifacts behind the adk_artifact_service behaviour. Artifact authority is an explicit scope:

{app, AppName}
{user, AppName, UserId}
{session, AppName, UserId, SessionId}

The same logical name in two scopes is two independent artifacts. A successful put always allocates a new positive version; deletion never makes that version reusable. The filesystem adapter also has an explicit lifetime allocation ceiling for scopes, names, and versions, described below. A service handle is passed to a Runner as {Module, Handle}. Provider code never receives an ETS table or filesystem root.

Capability and listing API

Adapters expose their versioned contract and configured bounds:

{ok, #{api_version := 1,
       immutable_versions := true,
       scopes := [app, user, session],
       pagination := Pagination,
       quotas := Quotas}} = adk_artifact_ets:capabilities(ArtifactPid).

Use list_names/3 to enumerate unique logical names and list_versions/4 to enumerate metadata-only versions. Both use an exclusive cursor and return at most limit entries:

{ok, #{scope := Scope, items := Names, next_cursor := NameCursor}} =
    adk_artifact_ets:list_names(
      ArtifactPid, Scope, #{limit => 100}),

{ok, #{items := Versions, next_cursor := VersionCursor}} =
    adk_artifact_ets:list_versions(
      ArtifactPid, Scope, <<"reports/summary.txt">>, #{limit => 100}),

NextNames = case NameCursor of
    undefined -> [];
    _ ->
        {ok, #{scope := Scope, items := Page}} = adk_artifact_ets:list_names(
            ArtifactPid, Scope, #{limit => 100, cursor => NameCursor}),
        Page
end.

Name cursors are the final binary name returned by the previous page. Version cursors are the final positive integer version returned. Results are sorted by name and then version. Version pages never contain the artifact data.

The original list/2 API remains for compatibility. It returns every version only when the result fits legacy_list_limit; otherwise it returns {error, result_limit_exceeded}. New code should use the distinct paginated operations.

put/6, get/5, and delete/5 add bounded call options while the original arities keep their existing defaults:

adk_artifact_ets:put(
  ArtifactPid, Scope, <<"result.txt">>, <<"ready">>,
  #{mime_type => <<"text/plain">>},
  #{timeout_ms => 2000}).

The absolute deadline calculated by the caller is carried with the request and is checked by the adapter immediately before a mutation is committed. If a request expires while queued, it returns {error, timeout} and is not silently committed after the caller has stopped waiting. A filesystem request that has already reserved a version may leave a gap, but it cannot publish an artifact after observing an expired deadline.

Shared validation limits

Both bundled adapters fail closed with the same structural limits:

  • each app name, user ID, and session ID is non-empty UTF-8, contains no NUL, and is at most 256 bytes;
  • an artifact name is non-empty UTF-8, contains no empty, . or .. path segment, contains no NUL, and is at most 1,024 bytes;
  • a MIME type is valid UTF-8 without CR, LF or NUL, contains /, and is at most 255 bytes;
  • custom metadata is a JSON-safe map with binary UTF-8 keys, at most 16,384 encoded bytes, 128 aggregate map/list entries, and eight levels of nesting;
  • call timeouts are positive and no greater than 300,000 milliseconds.

Unknown configuration, artifact, pagination, call, and repair options are rejected. capabilities/1 exposes these limits for tooling without requiring callers to duplicate constants.

In-memory ETS adapter

adk_artifact_ets is intended for tests and ephemeral nodes:

{ok, ArtifactPid} = adk_artifact_ets:start_link(
    #{max_artifact_bytes => 67108864,
      max_scope_bytes => 268435456,
      max_total_bytes => 536870912,
      max_scope_artifacts => 25000,
      max_total_artifacts => 100000,
      max_page_limit => 1000,
      legacy_list_limit => 1000}),
ArtifactService = {adk_artifact_ets, ArtifactPid}.

The adapter enforces item, per-scope, and service-wide byte and count quotas before inserting an artifact. Deletion releases byte/count capacity but keeps the high-water version counter. Quota failures are explicit, for example {error, artifact_too_large} or {error, {quota_exceeded, max_scope_bytes}}.

All data and counters disappear when the service process stops. The current reference adapter coordinates operations through one GenServer; deployments that require filesystem/object-store-scale independent-scope throughput should use a persistent adapter with sharded workers rather than increasing ETS quotas without bound.

Exact-scope sharded adapter

adk_artifact_sharded preserves the artifact-service API while assigning one stable supervised worker to each exact app, user, or session scope. Calls for one scope retain the selected adapter's ordering; after first resolution, callers reach that worker through a protected ETS fast path, so unrelated scopes do not queue behind the router or one storage GenServer:

{ok, ShardedArtifacts} = adk_artifact_sharded:start_link(
    #{adapter => adk_artifact_ets,
      adapter_config => #{max_artifact_bytes => 1048576},
      max_active_scopes => 1024,
      max_router_queue => 256}),
{ok, #{version := 1}} = adk_artifact_sharded:put(
    ShardedArtifacts,
    {session, <<"my_app">>, <<"user-42">>, <<"session-7">>},
    <<"reports/summary.txt">>, <<"ready">>,
    #{mime_type => <<"text/plain">>}),
{ok, #{active_scopes := 1, routing := exact_scope}} =
    adk_artifact_sharded:status(ShardedArtifacts),
ok = adk_artifact_sharded:stop(ShardedArtifacts).

The default worker adapter is ETS. For durable shards, select adapter => adk_artifact_fs and put the base root in adapter_config; the router derives a deterministic SHA-256, path-safe subroot for each scope, so a router restart reopens the same data. The owner process and explicit stop/1 both synchronously clean up the per-instance dynamic supervisor and children. A failed worker is removed and recreated on the next call; an ETS shard is volatile, while a filesystem shard reloads its published versions.

max_active_scopes and max_router_queue are hard admission bounds. An independent guard monitors each unresolved caller, owns the atomic cold-route permit, and releases it on caller death or timeout. The router rechecks that caller before starting a worker, so a stale queued request cannot create an abandoned shard. There is no idle-worker eviction in 0.5, so reaching the active-scope ceiling returns {error, max_active_scopes_reached} until a worker exits or the router is restarted. Adapter quotas are enforced independently inside each shard; capabilities deliberately report global_quota => false. Use an outer admission policy or a custom adapter when the deployment requires one aggregate budget across all scopes.

Durable filesystem adapter

adk_artifact_fs persists data and metadata below a dedicated root:

{ok, ArtifactPid} = adk_artifact_fs:start_link(
    #{root => <<"/var/lib/my_app/adk-artifacts">>,
      max_artifact_bytes => 67108864,
      max_page_limit => 1000,
      legacy_list_limit => 1000,
      max_scan_entries => 10000,
      recovery_grace_ms => 300000}),
ArtifactService = {adk_artifact_fs, ArtifactPid},

Scope = {session, <<"my_app">>, <<"user-42">>, <<"session-7">>},
{ok, #{version := 1}} = adk_artifact_fs:put(
    ArtifactPid, Scope, <<"reports/summary.txt">>, <<"ready">>,
    #{mime_type => <<"text/plain">>,
      metadata => #{<<"source">> => <<"agent">>}}),
{ok, #{data := <<"ready">>}} = adk_artifact_fs:get(
    ArtifactPid, Scope, <<"reports/summary.txt">>, latest).

The configured root must be a real directory, not a symbolic link. Scope and logical names are SHA-256-addressed and never become path components. Metadata is checked against the requested scope, name, and version on every read. The stored size and digest are checked before content is returned. Metadata and data files are size-checked before being read into the VM. Corruption fails as {error, corrupt_artifact} rather than returning untrusted bytes.

max_scan_entries must be at least three. To ensure allocation fails before the bounded directory scans themselves become unreadable, the adapter admits:

  • max_scan_entries div 2 lifetime scopes per storage root;
  • max_scan_entries div 2 lifetime logical names per scope; and
  • max_scan_entries div 3 lifetime versions per scoped logical name.

Capabilities expose these values as max_lifetime_scopes, max_lifetime_names_per_scope, and max_lifetime_versions_per_name in the quotas map. A new allocation beyond them returns {error, artifact_scope_capacity_reached}, {error, artifact_name_capacity_reached}, or {error, artifact_version_capacity_reached} respectively. Scope/name slots and version reservations are created exclusively, making admission safe across multiple service processes sharing one root. Choose the scan bound from lifetime churn, not only concurrently retained artifacts.

Publication and crash recovery

Each put creates an exclusively written, synced .reserve file first. The payload and metadata are then written and synced under request-unique staging names. The payload is renamed into place before the metadata; the atomic metadata rename is the only publication point. Readers and list operations only consider final metadata names, so they ignore an interrupted payload or partial staging file. Multiple service processes using the same root race on exclusive reservations and therefore allocate distinct versions.

The reference filesystem adapter still coordinates each configured service through one GenServer, so filesystem scans and mutations for independent scopes are serialized within that process. The reservation protocol keeps multiple service instances safe, but it is not a throughput-sharding policy. High-throughput deployments should partition scopes across supervised workers with adk_artifact_sharded, or implement the same service contract over a sharded object store.

A crash can leave staging files or a final payload without published metadata. They are invisible, and repair/2 can remove them after a safety grace period:

{ok, #{scanned := Scanned,
       removed := Removed,
       reservations_preserved := Preserved,
       corrupt := Corrupt}} =
    adk_artifact_fs:repair(
      ArtifactPid, #{limit => 1000, min_age_ms => 300000}).

Repair never removes reservation files, so an interrupted or deleted version is not reused after restart. It reports, but does not automatically erase, a published metadata record whose data file is missing. min_age_ms => 0 is useful in a stopped-writer maintenance window and deterministic tests; do not use it while another service instance may still be staging a write. Traversal and repair are bounded by max_scan_entries and the requested repair limit.

Because scope/name slots and version reservations are durable non-reuse tombstones, deleting artifacts does not restore any lifetime allocation capacity. Repair recognizes and preserves the slot records; it does not count them as ordinary staging garbage. Once a bound is reached, rotate to a new name, scope, or root as appropriate, or restart with a deliberately larger max_scan_entries. Focused filesystem coverage passes 15 tests, including multi-instance slot races, failure before listing/repair scan exhaustion, delete/restart persistence of every ceiling, publication, repair, corruption, and deadline cases.

Erlang's portable file API can sync each created file but does not expose a portable directory fsync. The adapter therefore guarantees atomic reader visibility and synced file contents; deployments requiring a documented power-loss directory-entry guarantee should provide an object-store adapter or a platform-specific storage layer with that durability contract.

The artifact directory is an application-owned trust boundary. Give it least-privilege filesystem permissions, keep it outside a served document root, and use an encrypted volume when artifact confidentiality requires it. The adapter does not encrypt content itself and must not be used as a secret store.

Runner integration

Pass a validated service reference to adk_runner:new/4:

Runner = adk_runner:new(
    AgentPid, <<"my_app">>, erlang_adk_session,
    #{artifact_svc => ArtifactService}).

Artifact instruction placeholders are resolved only from the exact invocation scope through a bounded service call. A local tool that declares artifact authority receives an opaque single-call capability, not ArtifactService:

-module(save_report_tool).
-behaviour(adk_tool).

-export([schema/0, context_capabilities/0, execute/2]).

schema() ->
    #{<<"name">> => <<"save_report">>,
      <<"description">> => <<"Save the generated report">>,
      <<"parameters">> =>
          #{<<"type">> => <<"object">>,
            <<"properties">> =>
                #{<<"text">> => #{<<"type">> => <<"string">>}},
            <<"required">> => [<<"text">>],
            <<"additionalProperties">> => false}}.

context_capabilities() -> [artifact_put].

execute(#{<<"text">> := Text}, Context) ->
    adk_context:save_artifact(
      Context, <<"reports/final.txt">>, Text,
      #{mime_type => <<"text/plain">>}).

The public helpers are save_artifact/4, load_artifact/3, list_artifacts/2, list_artifact_versions/3, delete_artifact/3, and attach_artifact/3. Corresponding declarations are artifact_put, artifact_get, artifact_list, artifact_list_versions, artifact_delete, and artifact_attach. An undeclared operation fails as {error, {context_capability_denied, Operation}}.

The opaque capability is also the adapter trust boundary. Successful put/get/version-list and legacy-list records must embed the exact bound scope and requested name; modern name-page envelopes also carry the exact scope and their bare names are shape/name validated. A foreign-scope, wrong-name, or malformed successful response fails closed as {error, invalid_artifact_service_reply} before data or an effect escapes.

Successful tool mutations are correlated to the tool call and projected into the canonical tool event's <<"context_effects">> action as metadata-only artifact deltas. Artifact data and service handles are never stored there.

Model-selected attachment

Add adk_load_artifacts_tool to an agent's tool list when the model should be able to select existing artifacts:

{ok, AgentPid} = erlang_adk:spawn_agent(
    <<"ArtifactReader">>,
    #{provider => adk_llm_gemini,
      model => <<"gemini-3.1-flash-lite">>,
      instructions =>
          <<"Load a relevant artifact before answering questions about it.">>},
    [adk_load_artifacts_tool]),
Runner = adk_runner:new(
    AgentPid, <<"my_app">>, erlang_adk_session,
    #{artifact_svc => ArtifactService,
      context_policy =>
          #{max_bytes => 1048576,
            max_request_bytes => 2097152,
            overflow => error}}).

The tool returns metadata only. Runner resolves the exact committed name/version, verifies scope, size, MIME type, and SHA-256 digest, and injects bounded adk_content parts into only the next model request. A later model round must request the artifact again. Bytes are not copied into session state, ordinary history, the durable tool response, or developer diagnostics.

The built-in supports at most eight selected artifacts and Runner applies a 10 MiB aggregate attachment bound. The underlying artifact service may impose a smaller item limit.

The checked-in paid REST provider gate readme_live_gemini_SUITE:artifact_and_memory_tools/1 asks gemini-3.1-flash-lite to select a real scoped image, proves the correlated attachment effect, and proves the bytes are absent from persisted session events. The targeted REST case passed on 2026-07-14. This supplements the deterministic scope, attachment, and persistence tests. The final v0.7 full REST run also passed this case; 15 of 17 REST cases passed overall, with the two failures caused by HTTP 429 in Search grounding and cached-content creation.

The built-in declaration contains strict JSON Schema constraints, including additionalProperties. At the Gemini boundary it is therefore emitted as parametersJsonSchema, while Erlang ADK retains the same compiled schema for local argument validation. The targeted REST pass exercises that projection.

Developer inspection

The authenticated developer API and CLI can list names and metadata-only versions and can delete an exact scope after a matching confirmation payload. They never return artifact bytes or custom metadata. Configure an exact-scope resource_provider as documented in the README; a provider receives the requested {session, App, User, Session} and may return {error, forbidden}. Each returned version must also embed that exact scope. A mismatched adapter record is rejected as unavailable instead of being relabelled under the HTTP path scope.

Current 0.5 limitations are explicit:

  • whole-binary operations are bounded, but credit/ack upload/download and byte-range streaming are not implemented;
  • the filesystem adapter provides atomic reader visibility and repairable staging but cannot promise a portable directory fsync through Erlang's cross-platform file API;
  • artifact event deltas are implemented, while full durable orphan recovery after a session-event persistence failure remains partial;
  • direct ETS and filesystem services serialize operations per process; the optional sharded wrapper provides bounded exact-scope overlap, but does not aggregate quotas, idle-evict workers, or expose a cross-shard repair/2 operation;
  • filesystem lifetime allocation is deliberately finite for scopes, names, and versions; deletion preserves non-reuse slots/reservations and therefore does not replenish capacity;
  • developer tooling deliberately supports inspection and deletion, not raw upload/download, filesystem repair, or content preview.