ExZarr.Storage.Backend.S3 (ExZarr v1.1.0)

View Source

AWS S3 storage backend for Zarr arrays.

Stores chunks and metadata in Amazon S3, providing scalable cloud storage with high availability and durability.

Configuration

Requires the following options:

  • :bucket - S3 bucket name (required)
  • :prefix - Key prefix/path within bucket (optional, default: "")
  • :region - AWS region (optional, default: "us-east-1")
  • :endpoint_url - Custom endpoint URL for S3-compatible services (optional)

AWS credentials are automatically loaded from standard AWS credential sources:

  • Environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
  • Shared credentials file (~/.aws/credentials)
  • IAM role (when running on EC2/ECS)

For testing with localstack or minio, set the AWS_ENDPOINT_URL environment variable:

export AWS_ENDPOINT_URL=http://localhost:4566  # localstack
export AWS_ENDPOINT_URL=http://localhost:9000  # minio

Dependencies

Requires ex_aws and ex_aws_s3 packages:

{:ex_aws, "~> 2.5"},
{:ex_aws_s3, "~> 2.5"}

Example

# Register the S3 backend
:ok = ExZarr.Storage.Registry.register(ExZarr.Storage.Backend.S3)

# Create array with S3 storage
{:ok, array} = ExZarr.create(
  shape: {1000, 1000},
  chunks: {100, 100},
  dtype: :float64,
  storage: :s3,
  bucket: "my-zarr-data",
  prefix: "experiments/array1"
)

# Write and read data
ExZarr.Array.set_slice(array, data, start: {0, 0}, stop: {100, 100})
{:ok, result} = ExZarr.Array.get_slice(array, start: {0, 0}, stop: {100, 100})

S3 Structure

Arrays are stored with the following key structure:

s3://bucket/prefix/.zarray           # Metadata
s3://bucket/prefix/0.0               # Chunk at index (0, 0)
s3://bucket/prefix/0.1               # Chunk at index (0, 1)

Performance Considerations

  • Chunks are read/written individually (parallel access recommended)
  • Consider chunk size vs. S3 request overhead
  • Use S3 Transfer Acceleration for global access
  • Configure appropriate IAM permissions

Error Handling

S3 errors are returned as {:error, reason} tuples with details from AWS. Common errors:

  • :bucket_not_found - Bucket doesn't exist
  • :access_denied - Insufficient permissions
  • :network_error - Network connectivity issues