ExZarr.Storage.Backend.GCS (ExZarr v1.1.0)

View Source

Google Cloud Storage (GCS) backend for Zarr arrays.

Stores chunks and metadata in Google Cloud Storage, providing globally distributed object storage with strong consistency.

Configuration

Requires the following options:

  • :bucket - GCS bucket name (required)
  • :prefix - Object prefix/path within bucket (optional, default: "")
  • :credentials - Path to service account JSON file or credentials map (required)
  • :endpoint_url - Custom endpoint URL for fake-gcs-server or compatible services (optional)

For testing with fake-gcs-server, set the GCS_ENDPOINT_URL environment variable:

export GCS_ENDPOINT_URL=http://localhost:4443  # fake-gcs-server

Dependencies

Requires the goth and req packages:

{:goth, "~> 1.4"},
{:req, "~> 0.4"}

Authentication

Uses Google Cloud service account credentials. Credentials can be provided:

  • As a path to a JSON key file: credentials: "/path/to/service-account.json"
  • As a decoded map: credentials: %{...}
  • Via GOOGLE_APPLICATION_CREDENTIALS environment variable

Example

# Register the GCS backend
:ok = ExZarr.Storage.Registry.register(ExZarr.Storage.Backend.GCS)

# Create array with GCS storage
{:ok, array} = ExZarr.create(
  shape: {1000, 1000},
  chunks: {100, 100},
  dtype: :float64,
  storage: :gcs,
  bucket: "my-zarr-bucket",
  prefix: "experiments/array1",
  credentials: System.get_env("GOOGLE_APPLICATION_CREDENTIALS")
)

# Write and read data
ExZarr.Array.set_slice(array, data, start: {0, 0}, stop: {100, 100})
{:ok, result} = ExZarr.Array.get_slice(array, start: {0, 0}, stop: {100, 100})

GCS Structure

Arrays are stored with the following object paths:

gs://bucket/prefix/.zarray           # Metadata
gs://bucket/prefix/0.0               # Chunk at index (0, 0)
gs://bucket/prefix/0.1               # Chunk at index (0, 1)

Performance Considerations

  • Objects are read/written individually
  • Use appropriate chunk sizes for your access patterns
  • Consider using Cloud CDN for read-heavy workloads
  • Configure appropriate storage classes (Standard/Nearline/Coldline/Archive)

Error Handling

GCS errors are returned as {:error, reason} tuples. Common errors:

  • :bucket_not_found - Bucket doesn't exist
  • :access_denied - Insufficient permissions
  • :network_error - Network connectivity issues