ExZarr.Storage (ExZarr v1.1.0)

View Source

Storage backend abstraction for Zarr arrays.

Provides a unified interface for storing and retrieving Zarr array data across different storage backends. Each backend handles chunks and metadata according to the Zarr v2 specification.

Available Backends

  • :memory - In-memory storage using Elixir maps. Fast but non-persistent. Suitable for temporary arrays and testing.

  • :filesystem - Local filesystem storage using the Zarr v2 directory structure. Chunks are stored as individual files with dot notation (e.g., 0.0). Metadata is stored in .zarray JSON files.

  • :zip - Zip archive storage. All chunks and metadata are stored in a single zip file. Useful for archiving, distribution, and reducing file count. Uses Erlang's built-in :zip module.

Zarr Directory Structure

For filesystem storage, arrays follow this structure:

/path/to/array/
  .zarray          # JSON metadata file
  0.0              # Chunk at index (0, 0)
  0.1              # Chunk at index (0, 1)
  1.0              # Chunk at index (1, 0)
  ...

Examples

# Initialize memory storage
{:ok, storage} = ExZarr.Storage.init(%{storage_type: :memory})

# Initialize filesystem storage
{:ok, storage} = ExZarr.Storage.init(%{
  storage_type: :filesystem,
  path: "/tmp/my_array"
})

# Write and read chunks
:ok = ExZarr.Storage.write_chunk(storage, {0, 0}, data)
{:ok, data} = ExZarr.Storage.read_chunk(storage, {0, 0})

# Write and read metadata
:ok = ExZarr.Storage.write_metadata(storage, metadata, [])
{:ok, metadata} = ExZarr.Storage.read_metadata(storage)

Summary

Functions

Deletes a chunk from storage.

Initializes a new storage backend.

Lists all chunk keys in the storage.

Opens an existing storage backend.

Reads a chunk from storage.

Reads metadata from storage.

Writes a chunk to storage.

Writes metadata to storage.

Types

backend()

@type backend() :: :memory | :filesystem | :zip | :http | :s3

t()

@type t() :: %ExZarr.Storage{backend: backend(), path: String.t() | nil, state: map()}

Functions

delete_chunk(storage, chunk_index)

@spec delete_chunk(t(), tuple()) :: :ok | {:error, term()}

Deletes a chunk from storage.

Removes a chunk from the storage backend. This is used when resizing arrays to shrink dimensions.

Parameters

  • storage - Storage instance
  • chunk_index - Tuple identifying the chunk to delete

Examples

# Delete a specific chunk
:ok = ExZarr.Storage.delete_chunk(storage, {0, 0})

Returns

  • :ok on success
  • {:error, reason} on failure

init(config)

@spec init(map()) :: {:ok, t()} | {:error, term()}

Initializes a new storage backend.

Creates a new storage instance for the specified backend type. For filesystem storage, creates the directory if it doesn't exist.

Parameters

  • config - Map with :storage_type and optional :path

Examples

# Memory storage
{:ok, storage} = ExZarr.Storage.init(%{storage_type: :memory})

# Filesystem storage
{:ok, storage} = ExZarr.Storage.init(%{
  storage_type: :filesystem,
  path: "/tmp/my_array"
})

Returns

  • {:ok, storage} on success
  • {:error, :path_required} if filesystem storage without path
  • {:error, :invalid_storage_config} for unsupported backends
  • {:error, {:mkdir_failed, reason}} if directory creation fails

list_chunks(storage)

@spec list_chunks(t()) :: {:ok, [tuple()]} | {:error, term()}

Lists all chunk keys in the storage.

Returns a list of all chunk indices that have been written to storage. For filesystem storage, reads the directory and parses chunk filenames. For memory storage, returns the keys from the chunks map.

Examples

{:ok, chunks} = ExZarr.Storage.list_chunks(storage)
# => [{0, 0}, {0, 1}, {1, 0}, {1, 1}]

Returns

  • {:ok, [chunk_indices]} with list of chunk index tuples
  • {:error, reason} on failure

Note

The order of chunks in the returned list is not guaranteed.

open(opts)

@spec open(keyword()) :: {:ok, t()} | {:error, term()}

Opens an existing storage backend.

Opens a previously created storage location, typically for loading an existing array. The storage must already exist (use init/1 to create new storage).

Options

  • :path - Path to the storage directory (required for filesystem)
  • :storage - Backend type (default: :filesystem)

Examples

# Open filesystem storage
{:ok, storage} = ExZarr.Storage.open(path: "/tmp/my_array")

# Open with explicit backend
{:ok, storage} = ExZarr.Storage.open(
  path: "/tmp/my_array",
  storage: :filesystem
)

Returns

  • {:ok, storage} on success
  • {:error, :path_not_found} if path does not exist
  • {:error, :cannot_open_memory_storage} for memory backend
  • {:error, :invalid_storage_backend} for unsupported backends

read_chunk(storage, chunk_index)

@spec read_chunk(t(), tuple()) :: {:ok, binary()} | {:error, term()}

Reads a chunk from storage.

Retrieves compressed chunk data from storage. The chunk must have been previously written.

Parameters

  • storage - Storage instance
  • chunk_index - Tuple identifying the chunk (e.g., {0, 0})

Examples

{:ok, data} = ExZarr.Storage.read_chunk(storage, {0, 0})
{:error, :not_found} = ExZarr.Storage.read_chunk(storage, {99, 99})

Returns

  • {:ok, binary} with compressed chunk data
  • {:error, :not_found} if chunk doesn't exist
  • {:error, reason} for other failures

read_metadata(storage)

@spec read_metadata(t()) ::
  {:ok, ExZarr.Metadata.t() | ExZarr.MetadataV3.t()} | {:error, term()}

Reads metadata from storage.

Loads array metadata from storage. Automatically detects Zarr format version (v2 or v3) and returns the appropriate metadata struct:

Converts JSON data types back to internal format (e.g., "<f8" to :float64 for v2, or "float64" to :float64 for v3).

Examples

# Reading v2 array
{:ok, metadata} = ExZarr.Storage.read_metadata(storage)
metadata.shape    # => {1000, 1000}
metadata.dtype    # => :float64
metadata.compressor  # => :zlib

# Reading v3 array
{:ok, metadata} = ExZarr.Storage.read_metadata(storage)
metadata.shape    # => {1000, 1000}
metadata.data_type  # => "float64"
metadata.codecs   # => [%{name: "bytes"}, %{name: "gzip"}]

Returns

  • {:ok, metadata} with parsed Metadata struct (v2) or MetadataV3 struct (v3)
  • {:error, :metadata_not_found} if metadata file doesn't exist
  • {:error, reason} for other failures

write_chunk(storage, chunk_index, data)

@spec write_chunk(t(), tuple(), binary()) :: :ok | {:error, term()}

Writes a chunk to storage.

Stores compressed chunk data in the storage backend. For filesystem storage, creates a file using dot notation (e.g., 0.0 for chunk {0, 0}). For memory storage, returns an updated storage struct.

Parameters

  • storage - Storage instance
  • chunk_index - Tuple identifying the chunk
  • data - Binary data to write (typically compressed)

Examples

# Filesystem storage
:ok = ExZarr.Storage.write_chunk(storage, {0, 0}, compressed_data)

# Memory storage (returns updated storage)
{:ok, new_storage} = ExZarr.Storage.write_chunk(storage, {0, 0}, data)

Returns

  • :ok for filesystem storage
  • {:ok, updated_storage} for memory storage
  • {:error, reason} on failure

write_metadata(storage, metadata, opts)

@spec write_metadata(t(), ExZarr.Metadata.t() | ExZarr.MetadataV3.t(), keyword()) ::
  :ok | {:error, term()}

Writes metadata to storage.

Saves array metadata to storage. Automatically handles both Zarr v2 and v3 formats:

  • v2: Saves to .zarray file, converts :float64 to "<f8" format
  • v3: Saves to zarr.json file, converts :float64 to "float64" format

Parameters

  • storage - Storage instance
  • metadata - Metadata struct (v2) or MetadataV3 struct (v3) to write
  • opts - Options (currently unused)

Examples

# Write v2 metadata
metadata = %ExZarr.Metadata{
  shape: {1000, 1000},
  chunks: {100, 100},
  dtype: :float64,
  compressor: :zlib,
  fill_value: 0.0,
  order: "C",
  zarr_format: 2
}
:ok = ExZarr.Storage.write_metadata(storage, metadata, [])

# Write v3 metadata
metadata = %ExZarr.MetadataV3{
  zarr_format: 3,
  node_type: :array,
  shape: {1000, 1000},
  data_type: "float64",
  chunk_grid: %{name: "regular", configuration: %{chunk_shape: {100, 100}}},
  chunk_key_encoding: %{name: "default"},
  codecs: [%{name: "bytes"}, %{name: "gzip", configuration: %{level: 5}}],
  fill_value: 0.0,
  attributes: %{}
}
:ok = ExZarr.Storage.write_metadata(storage, metadata, [])

Returns

  • :ok for filesystem storage
  • {:ok, updated_storage} for memory storage
  • {:error, reason} on failure