ALLM.Pipeline.Artifacts.Dynamo (allm_pipeline v0.1.0)

Copy Markdown View Source

DynamoDB client for artifact storage.

Table Schema

  • pk (String): Primary key, format ARTIFACT#<artifact_id>
  • sk (String): Sort key, always ARTIFACT
  • artifact_id (String): UUID matching step_log.id
  • content_type (String): MIME type (text/html, application/json, text/plain)
  • content (Binary): Artifact content (optionally gzip compressed)
  • compressed (Boolean): Whether content is gzip compressed
  • checksum (String): SHA-256 hash of original content
  • size_bytes (Number): Original uncompressed size
  • created_at (String): ISO8601 timestamp

Two surfaces

put/4 / fetch/1 / delete/1 / exists?/1 implement ALLM.Pipeline.Artifacts and speak in dynamo://<table>/<id> URLs — that is what ALLM.Pipeline.ArtifactStore calls. The *_artifact/* and table-admin functions beneath them are this module's own DynamoDB API, speaking in bare artifact ids; the mix tasks, DynamoCase and the eval harness use them directly to inspect and manage the table.

Summary

Functions

Is the local DynamoDB stack reachable?

Clear all artifacts from the table. Use with caution - primarily for testing.

Create the artifacts table in DynamoDB.

Delete the artifact behind a dynamo:// URL.

Delete an artifact from DynamoDB.

Delete the artifacts table from DynamoDB.

How many bytes content occupies inside a stored DynamoDB item.

The ExUnit tags to exclude when DynamoDB is unreachable, plus an operator hint.

Whether an artifact exists behind a dynamo:// URL.

Fetch the artifact behind a dynamo:// URL, as stored — still gzipped if it was stored gzipped. ALLM.Pipeline.ArtifactStore owns decompression.

Whether content fits in a DynamoDB item once stored.

Get an artifact from DynamoDB.

The largest encoded (post-base64) size fits_item?/1 admits — i.e. the ceiling encoded_size/1 is compared against, not a raw byte count.

Store an already-encoded payload and return its dynamo://<table>/<id> URL.

Check if the artifacts table exists.

Get the configured table name.

Types

artifact()

@type artifact() :: %{
  artifact_id: String.t(),
  content_type: String.t(),
  content: binary(),
  compressed: boolean(),
  checksum: String.t(),
  size_bytes: non_neg_integer(),
  created_at: String.t()
}

Functions

available?()

@spec available?() :: boolean()

Is the local DynamoDB stack reachable?

table_exists?/0 plus the rescue a test harness needs: a stack that is not merely missing the table but not listening at all raises out of ExAws rather than returning {:error, _}.

clear_table()

@spec clear_table() :: :ok | {:error, term()}

Clear all artifacts from the table. Use with caution - primarily for testing.

create_table()

@spec create_table() :: :ok | {:error, term()}

Create the artifacts table in DynamoDB.

delete(arg1)

@spec delete(ALLM.Pipeline.Artifacts.url()) :: :ok | {:error, term()}

Delete the artifact behind a dynamo:// URL.

delete_artifact(artifact_id)

@spec delete_artifact(String.t()) :: :ok | {:error, term()}

Delete an artifact from DynamoDB.

delete_table()

@spec delete_table() :: :ok | {:error, term()}

Delete the artifacts table from DynamoDB.

encoded_size(content)

@spec encoded_size(binary()) :: non_neg_integer()

How many bytes content occupies inside a stored DynamoDB item.

put_artifact/6 writes the body base64-encoded as a String attribute, so the stored form is 4 * ceil(n / 3) — a third larger than the binary handed in. DynamoDB's 400KB ceiling is an item-size limit on what is actually stored, so any size gate deciding "does this fit in DynamoDB?" must be applied to this number and not to the caller's original (or even its gzipped) byte count. See ALLM.Pipeline.ArtifactStore.store/4.

exclusions()

@spec exclusions() :: {[atom()], String.t() | nil}

The ExUnit tags to exclude when DynamoDB is unreachable, plus an operator hint.

Single source of truth for the tag list. Each consuming repo's own test_helper.exs and this repo's own need the same answer, because each suite starts ExUnit with its own option set, and since Phase 8 they live in DIFFERENT repos. Hand-copying the probe was survivable; hand-copying the TAG LIST is the "a rule enforced in more than one shape needs a membership guard" defect in its data-map form — add a third tag on one side and the other tree silently stops honouring it. This module is package lib/ code, so every suite calls the SAME implementation — that shared function is the cross-repo drift guard.

Why test-harness support sits in lib/ rather than test/support/: a test/support/ module is visible to exactly one repo's test tree, and this answer is needed by two or more — trees that, by design, cannot see each other's support code (the package declares no host dependency, and since Phase 8 each host is a different repo entirely). lib/ is the only place all can reach. Same reasoning, and the same precedent, as ALLM.Pipeline.Test shipping in lib/ (PHASE_1 §5.5). It does no I/O beyond the probe and is not referenced by any production path.

Returns {[], nil} when the stack is up (nothing excluded, nothing to print) and {tags, message} when it is down. The message is the caller's to print; this function does no I/O beyond the probe.

exists?(arg1)

@spec exists?(ALLM.Pipeline.Artifacts.url()) :: boolean()

Whether an artifact exists behind a dynamo:// URL.

A transport failure (as opposed to a missing item) has no clause and raises. That is unchanged pre-existing behaviour, deliberately preserved by the extraction: exists? has no way to say "I could not tell", and this phase changes no behaviour. Widening it to false would report a reachable artifact as absent whenever DynamoDB blinks. The behaviour's exists?/1 @callback doc names this adapter as the exception it allows.

A malformed URL is a different case and is NOT a raise — see artifact_id/1.

fetch(url)

@spec fetch(ALLM.Pipeline.Artifacts.url()) ::
  {:ok, ALLM.Pipeline.Artifacts.stored()} | {:error, term()}

Fetch the artifact behind a dynamo:// URL, as stored — still gzipped if it was stored gzipped. ALLM.Pipeline.ArtifactStore owns decompression.

fits_item?(content)

@spec fits_item?(binary()) :: boolean()

Whether content fits in a DynamoDB item once stored.

The whole capacity question lives here rather than in the caller: a tier router deciding "DynamoDB or S3?" should not have to know DynamoDB's item ceiling, its base64 encoding, or how much of the item its own metadata consumes.

content is the body as it will be handed to put_artifact/6 — i.e. already gzipped, if the caller compresses.

get_artifact(artifact_id)

@spec get_artifact(String.t()) :: {:ok, artifact()} | {:error, term()}

Get an artifact from DynamoDB.

max_payload_bytes()

@spec max_payload_bytes() :: pos_integer()

The largest encoded (post-base64) size fits_item?/1 admits — i.e. the ceiling encoded_size/1 is compared against, not a raw byte count.

A raw body of exactly this many bytes is therefore REFUSED; the largest admissible raw body is div(max_payload_bytes(), 4) * 3. Exposed so a boundary test can pin both edges of the metadata allowance without re-deriving the arithmetic — see dynamo_test.exs's fits_item?/1 describe, which derives the raw edge exactly that way.

put(id, content, content_type, map)

@spec put(
  ALLM.Pipeline.Artifacts.id(),
  binary(),
  String.t(),
  ALLM.Pipeline.Artifacts.meta()
) ::
  {:ok, ALLM.Pipeline.Artifacts.url()} | {:error, :too_large} | {:error, term()}

Store an already-encoded payload and return its dynamo://<table>/<id> URL.

The capacity check lives here rather than in the caller: a tier router deciding "DynamoDB or S3?" should not have to know DynamoDB's item ceiling, its base64 inflation, or how much of the item its own metadata consumes. An oversize payload short-circuits to {:error, :too_large} without contacting the server.

put_artifact(artifact_id, content, content_type, size_bytes, checksum, compressed)

@spec put_artifact(
  String.t(),
  binary(),
  String.t(),
  non_neg_integer(),
  String.t(),
  boolean()
) :: :ok | {:error, term()}

Store an artifact in DynamoDB.

table_exists?()

@spec table_exists?() :: boolean()

Check if the artifacts table exists.

table_name()

@spec table_name() :: String.t()

Get the configured table name.