ExZarr.ConsolidatedMetadata (ExZarr v1.1.0)

View Source

Consolidated metadata for Zarr groups and arrays.

Consolidated metadata stores all array and group metadata in a single .zmetadata file at the group root. This dramatically reduces the number of storage API calls needed to open a group, especially beneficial for cloud storage (S3, GCS) where API calls have significant latency and cost.

Benefits

  • Reduced API calls: Single read instead of N reads for N arrays
  • Faster group opening: ~90-99% reduction in cloud storage requests
  • Cost savings: Fewer S3/GCS API calls reduce costs
  • Atomic metadata: Consistent snapshot of all metadata

Format

The .zmetadata file contains a JSON object with:

{
  "zarr_consolidated_format": 1,
  "metadata": {
    ".zattrs": {...},
    ".zgroup": {...},
    "array1/.zarray": {...},
    "array1/.zattrs": {...},
    "array2/.zarray": {...},
    "array2/.zattrs": {...}
  }
}

Usage

# Consolidate metadata for a group
{:ok, _count} = ExZarr.ConsolidatedMetadata.consolidate(
  path: "/path/to/group",
  storage: :filesystem
)

# Check if consolidated metadata exists
true = ExZarr.ConsolidatedMetadata.exists?(
  path: "/path/to/group",
  storage: :filesystem
)

# Read consolidated metadata
{:ok, metadata_map} = ExZarr.ConsolidatedMetadata.read(
  path: "/path/to/group",
  storage: :filesystem
)

Compatibility

  • Compatible with zarr-python consolidated metadata
  • Works with both Zarr v2 and v3 formats
  • Optional feature - arrays work fine without it

Summary

Types

consolidate_opts()
metadata_map()

Functions

consolidate(opts)

Consolidate metadata for a Zarr group.

exists?(opts)

Check if consolidated metadata exists for a group.

read(opts)

Read consolidated metadata from a group.

remove(opts)

Remove consolidated metadata from a group.

Types

consolidate_opts()

@type consolidate_opts() :: [path: String.t(), storage: atom(), recursive: boolean()]

metadata_map()

@type metadata_map() :: %{required(String.t()) => map()}

Functions

consolidate(opts)

@spec consolidate(consolidate_opts()) :: {:ok, non_neg_integer()} | {:error, term()}

Consolidate metadata for a Zarr group.

Reads all array and group metadata files within a group and writes them to a single .zmetadata file.

Options

  • :path - Path to the group directory (required)
  • :storage - Storage backend atom (default: :filesystem)
  • :recursive - Include nested groups (default: true)

Returns

  • {:ok, count} - Number of metadata entries consolidated
  • {:error, reason} - Consolidation failed

Examples

# Consolidate a group
{:ok, 5} = ExZarr.ConsolidatedMetadata.consolidate(
  path: "/data/my_group",
  storage: :filesystem
)

# Non-recursive consolidation
{:ok, 2} = ExZarr.ConsolidatedMetadata.consolidate(
  path: "/data/my_group",
  storage: :filesystem,
  recursive: false
)

exists?(opts)

@spec exists?(keyword()) :: boolean()

Check if consolidated metadata exists for a group.

Options

  • :path - Path to the group directory (required)
  • :storage - Storage backend atom (default: :filesystem)

Returns

  • true - Consolidated metadata exists
  • false - No consolidated metadata

Examples

if ExZarr.ConsolidatedMetadata.exists?(path: "/data/my_group") do
  # Use consolidated metadata for faster access
end

read(opts)

@spec read(keyword()) :: {:ok, metadata_map()} | {:error, term()}

Read consolidated metadata from a group.

Returns a map of metadata entries keyed by their relative paths.

Options

  • :path - Path to the group directory (required)
  • :storage - Storage backend atom (default: :filesystem)

Returns

  • {:ok, metadata_map} - Map of relative_path => metadata
  • {:error, :not_found} - No consolidated metadata exists
  • {:error, reason} - Read failed

Examples

{:ok, metadata} = ExZarr.ConsolidatedMetadata.read(
  path: "/data/my_group",
  storage: :filesystem
)

# Access specific array metadata
array_meta = metadata["array1/.zarray"]

remove(opts)

@spec remove(keyword()) :: :ok | {:error, atom()}

Remove consolidated metadata from a group.

Deletes the .zmetadata file. Individual metadata files are not affected.

Options

  • :path - Path to the group directory (required)
  • :storage - Storage backend atom (default: :filesystem)

Returns

  • :ok - Consolidated metadata removed or didn't exist
  • {:error, reason} - Removal failed