Plugins, observability, and evaluation

This document defines the Erlang ADK 0.7.0 cross-cutting runtime contracts. It tracks the externally observable behavior of the official plugin, observability, and evaluation capability families while preserving Erlang's process-isolation and supervision model.

Runner plugin lifecycle

Plugins belong to a Runner, not an agent. adk_runner:new/4 compiles the ordered descriptor list once and rejects malformed configuration before a run is accepted. Each stateless hook executes in its own monitored lightweight process with descriptor-level timeout_ms, max_heap_words, and result-size bounds.

For a corresponding lifecycle phase, precedence is:

1. ordered Runner-global plugins;
2. the agent's existing local callback, unless a global plugin completed the phase early;
3. the model, tool, agent, or run operation;
4. ordered global after/error plugins;
5. the corresponding local after/error callback, unless a global plugin completed the phase early.

Outcomes

The same intervention vocabulary applies to the recoverable on_model_error and on_tool_error phases. on_agent_error, on_run_error, and success-only after_run are best-effort notifications: failures are recorded structurally, but neither a callback failure nor an intervention result can replace or halt the outcome being reported, even when the descriptor's ordinary failure policy is closed. The legacy on_error hook remains supported as a compatibility fallback.

failure_policy => open records a bounded structural trace and continues with the previous value. failure_policy => closed returns a typed, secret-free error. Exceptions, worker death, timeout, invalid results, oversized results, and observer attempts to intervene are all handled by that policy.

on_event runs before persistence. An amended event is accepted only when it preserves the event ID, invocation ID, author, actions/state delta, continuations, partial/final flags, and content kind. Final content cannot be rewritten after output-schema validation. This keeps plugins useful for policy and presentation without allowing them to bypass durable state or schema boundaries.

Direct erlang_adk:prompt/2 compatibility calls do not inherit Runner-global plugins. Agent-local callbacks continue to work there.

Stateful plugins

Stateful plugins implement adk_stateful_plugin and run behind one supervised adk_plugin_instance actor per descriptor. The adapter serializes callbacks through that actor, so plugin state changes are ordered even when Runner cases are concurrent. State commits only while the callback remains within its deadline and the owning process is alive; a timed-out or abandoned callback cannot install late state. The descriptor still applies bounded queue, heap, timeout, and result-size policy. Initialization also runs in a separate timeout/heap-bounded worker. The returned PID is the stable identity and its supervisor child is temporary: a crash does not silently replace it with an empty-state process behind a stale reference. Explicit recreation or an application persistence adapter is required for restart/durable state.

Built-ins

0.7 includes four opt-in plugins:

• adk_plugin_global_instruction adds a Runner-wide instruction through an amend-and-continue hook;
• adk_plugin_context_filter applies bounded context policy before model I/O;
• adk_plugin_reflect_retry converts bounded tool failures into explicit model-visible retry guidance;
• adk_plugin_metadata_logger records structural metadata without prompt, response, argument, result, media, credential, or token content.

They are ordinary descriptors and obey the same ordering, ownership, limits, and failure policy as application plugins.

Correlated observability

0.7 retains the legacy schema-version-1 lifecycle envelope and adds schema-version-2 operation signals at the actual model, tool, and Gemini Live boundaries. Operation spans carry nanosecond timing and duration. Correlation metadata includes W3C trace identifiers, run/invocation/session identifiers, agent/model, and tool/call identifiers where applicable.

adk_trace_context strictly parses and formats W3C traceparent and tracestate. Invalid, all-zero, oversized, duplicate, or malformed context is rejected rather than partly accepted. New child spans retain the caller's trace and sampling decision while receiving a fresh span ID.

The semantic mapping is pinned to gen-ai-semconv-development-2026-07-14. It is deliberately metadata-only: prompt/response text, tool arguments/results, audio/video bytes, thought signatures, authorization data, provider tokens, and API keys are not semantic attributes. This is true even when older lifecycle capture is explicitly enabled.

Metrics and delivery

adk_observability_metrics maintains a fixed instrument catalog and a bounded number of label series. New high-cardinality combinations overflow into a bounded aggregate rather than growing ETS state without limit.

Runner observability supports two delivery modes:

• synchronous exporters execute in descriptor order in monitored, timeout/heap-limited workers;
• delivery => async submits to a supervised adk_observability_bus with bounded item/byte/batch queues, retry and backoff, and explicit drop accounting.

The asynchronous bus provides bounded-best-effort delivery. An exporter must be idempotent because a delayed retry can repeat a batch; exhausted retries are dropped and counted. The queue, delayed-retry reservation, in-flight batches, and expiring owner-monitored drain waiters are all capped. Per-run exporter descriptors are not accepted in asynchronous mode because the long-lived bus owns its exporter configuration.

adk_otlp_http_json_exporter exports schema-v2 completed spans to the OTLP HTTP JSON trace endpoint and schema-v1 lifecycle records as logs. Span-start signals are not sent independently. The exporter enforces an explicit HTTP(S) origin/private-host policy, header/body limits, request timeout, and no redirects. It performs one HTTP attempt; retry belongs to the supervised asynchronous bus. The exporter classifies bounded failures as transient or permanent; the bus retries only transient failures and accounts permanent failures without looping.

The authenticated developer endpoint exposes only bounded operational snapshots and metadata. It is not a prompt, media, tool-payload, or trace archive.

Evaluation v2

adk_eval_set persists schema-version-2 evaluation sets and results. A case contains ordered turns and may run multiple samples. Cases can run concurrently up to concurrency; samples can run concurrently up to sample_concurrency; turns inside one conversation remain sequential.

The built-in criteria support:

• exact response matching;
• tool trajectory matching in exact, in_order, any_order, or subset mode;
• tool argument comparison in exact, subset, or ignored mode.

Criteria have explicit thresholds, minimum successful-sample requirements, and strict numeric/size bounds. An explicitly empty criteria list is an error; the adk eval run CLI chooses the exact-response criterion when --criteria is omitted.

adk_eval_agent_adapter creates a fresh agent, Runner, guardian, and session for each case/sample, and tears them down after completion. This is the default isolation path for an Erlang agent evaluation. Custom adk_eval_adapter implementations can target deterministic fixtures or remote systems, but they must preserve the same bounded output/event/trajectory contract.

Results include per-turn and per-sample outcomes, aggregate criteria, pass rate, thresholds, dataset revision, duration, and bounded caller build metadata. adk_eval_report renders JSON or Markdown and compares a candidate with a saved baseline using pass-drop and per-metric tolerance policy. Saved sets/results are checked, secret-pruned, and JSON round-trippable.

adk_eval_llm_judge is the first-party, explicit full-case rubric judge. Its metric descriptor uses kind => judge, scope => 'case', and module => adk_eval_llm_judge. Configuration requires binary rubric, rubric_id, and rubric_version; it defaults to adk_llm_gemini with gemini-3.1-flash-lite. The adapter forces a bounded structured-JSON response schema and validates an exact {score, rationale} object with score in 0..1. Prompt, output, rationale, token, timeout, and provider-worker heap bounds are finite. A monitored request worker dies on timeout or evaluation owner death and counts shared binaries in its heap ceiling. Independent sample workers continue to judge concurrently; no global judge server exists.

Provider modules and provider_config are trusted Erlang-only injection points for applications or deterministic tests and must satisfy the normal adk_llm provider contract. They are not accepted from the CLI criteria JSON. Provider credentials are not placed in the prompt or successful metadata; secret-bearing case fields are pruned, sensitive provider-config values are redacted from rationale, and raw provider failures are reduced to structural errors. Rationale remains evaluation content and is persisted in reports, so it must be protected under the case-data policy. The judge is never enabled implicitly and every call has provider cost/latency.

The non-interactive CLI entry point is:

adk eval run --config AGENT.json --eval-set SET.json \
  [--criteria CRITERIA.json] [--baseline BASELINE.json] \
  [--samples N] [--concurrency N] [--sample-concurrency N] \
  [--format json|markdown] [--output REPORT]

The command exits 0 when the candidate passes, 2 when evaluation completes but fails criteria or regression policy, and 1 for configuration/runtime errors.

Verification map

• adk_plugin_pipeline_test and adk_plugin_runner_integration_test cover ordering, amend/return compatibility, phase-specific errors, local/global precedence, intervention, failure policy, limits, and event invariants.
• adk_plugin_builtin_test and adk_plugin_stateful_test cover built-ins, actor serialization, deadline fencing, owner death, and state isolation.
• adk_observability_test, adk_observability_v2_test, adk_observability_runner_test, and adk_trace_context_test cover legacy compatibility, actual operation spans, semantic attributes, bounded metrics, synchronous/asynchronous delivery, and strict W3C propagation.
• adk_otlp_json_test and adk_otlp_http_json_exporter_test cover OTLP JSON, endpoint/header/body policy, redirects, timeouts, retry ownership, and metadata-only export.
• adk_live_observability_test covers Gemini Live connect/receive/tool operation signals.
• adk_eval_set_test, adk_eval_criteria_test, adk_eval_v2_test, adk_eval_llm_judge_test, and adk_eval_dev_view_test cover v2 validation, sampling/concurrency, fresh-runtime isolation, built-in criteria, bounded rubric judging, reports, and baselines. The opt-in REST suite contains a real gemini-3.1-flash-lite rubric-judge case.
• adk_cli_test covers adk eval run, developer observability/Live commands, exit statuses, bounds, and structured connection failures.
• readme_examples_test compiles and runs the stateless example plugin, exporter, direct-agent adapter, and evaluation metric shown in the README; the verification commands compile the v0.7 stateful and Live-executor modules with warnings treated as errors.