ExZarr.Storage.Backend.Zip (ExZarr v1.1.0)

View Source

Zip archive storage backend for Zarr arrays.

Stores chunks and metadata in a single zip file, providing a compact, portable format for archiving and distributing Zarr arrays.

Architecture

Uses an in-memory cache (Agent) for reads and writes, with deferred disk writes. The zip file is only written when metadata is saved, reducing I/O overhead.

Zip Structure

array.zip
  ├── .zarray          # JSON metadata
  ├── 0.0              # Chunk at index {0, 0}
  ├── 0.1              # Chunk at index {0, 1}
  └── ...

Characteristics

  • Compact: Single file, compressed storage
  • Portable: Easy to share and distribute
  • Persistent: Data survives process restarts
  • Efficient: Reduces file count for arrays with many chunks
  • Read-optimized: Fast random access to chunks

Configuration

Requires a :path option specifying the zip file location:

{:ok, array} = ExZarr.create(
  shape: {1000, 1000},
  dtype: :float64,
  storage: :zip,
  path: "/tmp/my_array.zip"
)

Usage Pattern

  1. Create: Initialize in-memory cache
  2. Write: Chunks go to cache (fast)
  3. Save: Call save to write zip file to disk
  4. Open: Load entire zip into cache
  5. Read: Fast access from cache

Important Notes

  • Metadata save triggers disk write of entire zip
  • All chunks kept in memory for fast access
  • Not suitable for very large arrays (memory constraints)
  • Write operations require calling ExZarr.save/2 to persist