Local-disk ALLM.Pipeline.Artifacts adapter — the one that lets a fresh
clone run pipelines with zero cloud infrastructure.
Configure it and no DynamoDB (nor docker-compose) is needed for artifacts
to store, round-trip and render in the review UI:
config :allm_pipeline, ALLM.Pipeline.Artifacts,
impl: ALLM.Pipeline.Artifacts.Filesystem
config :allm_pipeline, ALLM.Pipeline.Artifacts.Filesystem,
root: "/var/tmp/my-artifacts"That first key still works on a host that declares an
ALLM.Pipeline.Registry: the registry's artifacts: supplies the DEFAULT and
a config-file impl: overrides it per environment (install/0 writes the
seam keys with put_new for exactly this promise — see that module's
"Precedence", pinned by registry_test.exs). Put it in config/dev.exs and
the production artifacts: declaration is untouched.
:root defaults to allm_pipeline_artifacts under the system temp
directory, and is read at RUNTIME on every call.
Layout and URLs
Each artifact is two files under :root, named after the URL-encoded id —
ids are not path-safe ("<step_log_id>:llm" is a real one), so
URI.encode_www_form/1 runs before anything touches the filesystem:
<root>/<encoded-id>— the payload, byte-for-byte asput/4received it. Still gzipped ifArtifactStoregzipped it, which is why a barecatof a compressed artifact shows binary;ArtifactStore.fetch/1is what decompresses.<root>/<encoded-id>.meta.json—content_type,checksum,size_bytes(the ORIGINAL, pre-gzip size),compressed,stored_at.
A URL is file:// plus the payload file's absolute path, so an operator
holding a step_logs.artifact_url can find the bytes without this module.
The sidecar is load-bearing, and is written FIRST
compressed lives only in the sidecar, and ArtifactStore.fetch/1 gunzips
only when it says true — so a payload with no readable sidecar cannot be
decoded, it can only be mis-decoded. Two consequences, both deliberate:
put/4writes the sidecar before the payload. A failed write therefore strands at worst an orphaned sidecar, which is inert —fetch/1answers{:error, :not_found},exists?/1isfalse, andgc/1never counts it (payload_paths/1rejects.meta.json). The reverse order would strand a payload thatexists?reports andfetch/1hands back as an undecompressed gzip stream inside an{:ok, _}tuple.fetch/1answers{:error, :missing_artifact_metadata}when the sidecar is absent or unparseable, rather than assumingcompressed: false. The bytes are still on disk for an operator to recover by hand; what this adapter will not do is present them as the artifact.
Moving :root invalidates existing URLs
fetch/1 refuses a path outside the currently-configured :root
({:error, :outside_artifact_root}) — which both closes the traversal a
path-carrying URL would otherwise open, and turns "I moved the root and old
artifacts silently vanished" into a named error. Re-point :root at the old
directory, or accept that pre-move artifacts are unreachable.
No size ceiling
put/4 never returns {:error, :too_large}; a filesystem has no item limit
to enforce. So a payload the DynamoDB adapter would refuse is written here
without comment, and this adapter cannot exercise ArtifactStore's oversize
routing.
Summary
Functions
@spec gc(keyword()) :: {:ok, non_neg_integer()}
Delete every artifact under :root last modified at or before
opts[:older_than] (a DateTime, default: now), returning how many payloads
went. With no options this empties the root.
Compares filesystem mtime, whose resolution is one second — hence "at or before" rather than "before", so an artifact written in the same second as the cutoff is collected rather than surviving a full purge.
The count is of payloads actually removed, not of payloads selected: a
payload whose File.rm/1 fails (permissions, a concurrent gc/1 that got
there first) is not counted.
@spec root() :: Path.t()
The directory artifacts are written under, resolved at runtime.
Public so an operator (or a test's teardown) can find the tree without re-deriving the default.