ALLM.Pipeline.Artifacts.Filesystem (allm_pipeline v0.1.0)

Copy Markdown View Source

Local-disk ALLM.Pipeline.Artifacts adapter — the one that lets a fresh clone run pipelines with zero cloud infrastructure.

Configure it and no DynamoDB (nor docker-compose) is needed for artifacts to store, round-trip and render in the review UI:

config :allm_pipeline, ALLM.Pipeline.Artifacts,
  impl: ALLM.Pipeline.Artifacts.Filesystem

config :allm_pipeline, ALLM.Pipeline.Artifacts.Filesystem,
  root: "/var/tmp/my-artifacts"

That first key still works on a host that declares an ALLM.Pipeline.Registry: the registry's artifacts: supplies the DEFAULT and a config-file impl: overrides it per environment (install/0 writes the seam keys with put_new for exactly this promise — see that module's "Precedence", pinned by registry_test.exs). Put it in config/dev.exs and the production artifacts: declaration is untouched.

:root defaults to allm_pipeline_artifacts under the system temp directory, and is read at RUNTIME on every call.

Layout and URLs

Each artifact is two files under :root, named after the URL-encoded id — ids are not path-safe ("<step_log_id>:llm" is a real one), so URI.encode_www_form/1 runs before anything touches the filesystem:

  • <root>/<encoded-id> — the payload, byte-for-byte as put/4 received it. Still gzipped if ArtifactStore gzipped it, which is why a bare cat of a compressed artifact shows binary; ArtifactStore.fetch/1 is what decompresses.
  • <root>/<encoded-id>.meta.jsoncontent_type, checksum, size_bytes (the ORIGINAL, pre-gzip size), compressed, stored_at.

A URL is file:// plus the payload file's absolute path, so an operator holding a step_logs.artifact_url can find the bytes without this module.

The sidecar is load-bearing, and is written FIRST

compressed lives only in the sidecar, and ArtifactStore.fetch/1 gunzips only when it says true — so a payload with no readable sidecar cannot be decoded, it can only be mis-decoded. Two consequences, both deliberate:

  • put/4 writes the sidecar before the payload. A failed write therefore strands at worst an orphaned sidecar, which is inert — fetch/1 answers {:error, :not_found}, exists?/1 is false, and gc/1 never counts it (payload_paths/1 rejects .meta.json). The reverse order would strand a payload that exists? reports and fetch/1 hands back as an undecompressed gzip stream inside an {:ok, _} tuple.
  • fetch/1 answers {:error, :missing_artifact_metadata} when the sidecar is absent or unparseable, rather than assuming compressed: false. The bytes are still on disk for an operator to recover by hand; what this adapter will not do is present them as the artifact.

Moving :root invalidates existing URLs

fetch/1 refuses a path outside the currently-configured :root ({:error, :outside_artifact_root}) — which both closes the traversal a path-carrying URL would otherwise open, and turns "I moved the root and old artifacts silently vanished" into a named error. Re-point :root at the old directory, or accept that pre-move artifacts are unreachable.

No size ceiling

put/4 never returns {:error, :too_large}; a filesystem has no item limit to enforce. So a payload the DynamoDB adapter would refuse is written here without comment, and this adapter cannot exercise ArtifactStore's oversize routing.

Summary

Functions

Delete every artifact under :root last modified at or before opts[:older_than] (a DateTime, default: now), returning how many payloads went. With no options this empties the root.

The directory artifacts are written under, resolved at runtime.

Functions

gc(opts \\ [])

@spec gc(keyword()) :: {:ok, non_neg_integer()}

Delete every artifact under :root last modified at or before opts[:older_than] (a DateTime, default: now), returning how many payloads went. With no options this empties the root.

Compares filesystem mtime, whose resolution is one second — hence "at or before" rather than "before", so an artifact written in the same second as the cutoff is collected rather than surviving a full purge.

The count is of payloads actually removed, not of payloads selected: a payload whose File.rm/1 fails (permissions, a concurrent gc/1 that got there first) is not counted.

root()

@spec root() :: Path.t()

The directory artifacts are written under, resolved at runtime.

Public so an operator (or a test's teardown) can find the tree without re-deriving the default.