ExZarr.Storage (ExZarr v1.1.0)
View SourceStorage backend abstraction for Zarr arrays.
Provides a unified interface for storing and retrieving Zarr array data across different storage backends. Each backend handles chunks and metadata according to the Zarr v2 specification.
Available Backends
:memory- In-memory storage using Elixir maps. Fast but non-persistent. Suitable for temporary arrays and testing.:filesystem- Local filesystem storage using the Zarr v2 directory structure. Chunks are stored as individual files with dot notation (e.g.,0.0). Metadata is stored in.zarrayJSON files.:zip- Zip archive storage. All chunks and metadata are stored in a single zip file. Useful for archiving, distribution, and reducing file count. Uses Erlang's built-in:zipmodule.
Zarr Directory Structure
For filesystem storage, arrays follow this structure:
/path/to/array/
.zarray # JSON metadata file
0.0 # Chunk at index (0, 0)
0.1 # Chunk at index (0, 1)
1.0 # Chunk at index (1, 0)
...Examples
# Initialize memory storage
{:ok, storage} = ExZarr.Storage.init(%{storage_type: :memory})
# Initialize filesystem storage
{:ok, storage} = ExZarr.Storage.init(%{
storage_type: :filesystem,
path: "/tmp/my_array"
})
# Write and read chunks
:ok = ExZarr.Storage.write_chunk(storage, {0, 0}, data)
{:ok, data} = ExZarr.Storage.read_chunk(storage, {0, 0})
# Write and read metadata
:ok = ExZarr.Storage.write_metadata(storage, metadata, [])
{:ok, metadata} = ExZarr.Storage.read_metadata(storage)
Summary
Functions
Deletes a chunk from storage.
Initializes a new storage backend.
Lists all chunk keys in the storage.
Opens an existing storage backend.
Reads a chunk from storage.
Reads metadata from storage.
Writes a chunk to storage.
Writes metadata to storage.
Types
Functions
Deletes a chunk from storage.
Removes a chunk from the storage backend. This is used when resizing arrays to shrink dimensions.
Parameters
storage- Storage instancechunk_index- Tuple identifying the chunk to delete
Examples
# Delete a specific chunk
:ok = ExZarr.Storage.delete_chunk(storage, {0, 0})Returns
:okon success{:error, reason}on failure
Initializes a new storage backend.
Creates a new storage instance for the specified backend type. For filesystem storage, creates the directory if it doesn't exist.
Parameters
config- Map with:storage_typeand optional:path
Examples
# Memory storage
{:ok, storage} = ExZarr.Storage.init(%{storage_type: :memory})
# Filesystem storage
{:ok, storage} = ExZarr.Storage.init(%{
storage_type: :filesystem,
path: "/tmp/my_array"
})Returns
{:ok, storage}on success{:error, :path_required}if filesystem storage without path{:error, :invalid_storage_config}for unsupported backends{:error, {:mkdir_failed, reason}}if directory creation fails
Lists all chunk keys in the storage.
Returns a list of all chunk indices that have been written to storage. For filesystem storage, reads the directory and parses chunk filenames. For memory storage, returns the keys from the chunks map.
Examples
{:ok, chunks} = ExZarr.Storage.list_chunks(storage)
# => [{0, 0}, {0, 1}, {1, 0}, {1, 1}]Returns
{:ok, [chunk_indices]}with list of chunk index tuples{:error, reason}on failure
Note
The order of chunks in the returned list is not guaranteed.
Opens an existing storage backend.
Opens a previously created storage location, typically for loading an
existing array. The storage must already exist (use init/1 to create new
storage).
Options
:path- Path to the storage directory (required for filesystem):storage- Backend type (default::filesystem)
Examples
# Open filesystem storage
{:ok, storage} = ExZarr.Storage.open(path: "/tmp/my_array")
# Open with explicit backend
{:ok, storage} = ExZarr.Storage.open(
path: "/tmp/my_array",
storage: :filesystem
)Returns
{:ok, storage}on success{:error, :path_not_found}if path does not exist{:error, :cannot_open_memory_storage}for memory backend{:error, :invalid_storage_backend}for unsupported backends
Reads a chunk from storage.
Retrieves compressed chunk data from storage. The chunk must have been previously written.
Parameters
storage- Storage instancechunk_index- Tuple identifying the chunk (e.g.,{0, 0})
Examples
{:ok, data} = ExZarr.Storage.read_chunk(storage, {0, 0})
{:error, :not_found} = ExZarr.Storage.read_chunk(storage, {99, 99})Returns
{:ok, binary}with compressed chunk data{:error, :not_found}if chunk doesn't exist{:error, reason}for other failures
@spec read_metadata(t()) :: {:ok, ExZarr.Metadata.t() | ExZarr.MetadataV3.t()} | {:error, term()}
Reads metadata from storage.
Loads array metadata from storage. Automatically detects Zarr format version (v2 or v3) and returns the appropriate metadata struct:
- v2: Loads
.zarrayfile, returnsExZarr.Metadatastruct - v3: Loads
zarr.jsonfile, returnsExZarr.MetadataV3struct
Converts JSON data types back to internal format (e.g., "<f8" to :float64
for v2, or "float64" to :float64 for v3).
Examples
# Reading v2 array
{:ok, metadata} = ExZarr.Storage.read_metadata(storage)
metadata.shape # => {1000, 1000}
metadata.dtype # => :float64
metadata.compressor # => :zlib
# Reading v3 array
{:ok, metadata} = ExZarr.Storage.read_metadata(storage)
metadata.shape # => {1000, 1000}
metadata.data_type # => "float64"
metadata.codecs # => [%{name: "bytes"}, %{name: "gzip"}]Returns
{:ok, metadata}with parsed Metadata struct (v2) or MetadataV3 struct (v3){:error, :metadata_not_found}if metadata file doesn't exist{:error, reason}for other failures
Writes a chunk to storage.
Stores compressed chunk data in the storage backend. For filesystem storage,
creates a file using dot notation (e.g., 0.0 for chunk {0, 0}). For
memory storage, returns an updated storage struct.
Parameters
storage- Storage instancechunk_index- Tuple identifying the chunkdata- Binary data to write (typically compressed)
Examples
# Filesystem storage
:ok = ExZarr.Storage.write_chunk(storage, {0, 0}, compressed_data)
# Memory storage (returns updated storage)
{:ok, new_storage} = ExZarr.Storage.write_chunk(storage, {0, 0}, data)Returns
:okfor filesystem storage{:ok, updated_storage}for memory storage{:error, reason}on failure
@spec write_metadata(t(), ExZarr.Metadata.t() | ExZarr.MetadataV3.t(), keyword()) :: :ok | {:error, term()}
Writes metadata to storage.
Saves array metadata to storage. Automatically handles both Zarr v2 and v3 formats:
- v2: Saves to
.zarrayfile, converts:float64to"<f8"format - v3: Saves to
zarr.jsonfile, converts:float64to"float64"format
Parameters
storage- Storage instancemetadata- Metadata struct (v2) or MetadataV3 struct (v3) to writeopts- Options (currently unused)
Examples
# Write v2 metadata
metadata = %ExZarr.Metadata{
shape: {1000, 1000},
chunks: {100, 100},
dtype: :float64,
compressor: :zlib,
fill_value: 0.0,
order: "C",
zarr_format: 2
}
:ok = ExZarr.Storage.write_metadata(storage, metadata, [])
# Write v3 metadata
metadata = %ExZarr.MetadataV3{
zarr_format: 3,
node_type: :array,
shape: {1000, 1000},
data_type: "float64",
chunk_grid: %{name: "regular", configuration: %{chunk_shape: {100, 100}}},
chunk_key_encoding: %{name: "default"},
codecs: [%{name: "bytes"}, %{name: "gzip", configuration: %{level: 5}}],
fill_value: 0.0,
attributes: %{}
}
:ok = ExZarr.Storage.write_metadata(storage, metadata, [])Returns
:okfor filesystem storage{:ok, updated_storage}for memory storage{:error, reason}on failure