ExZarr.MetadataV3 (ExZarr v1.1.0)

View Source

Zarr v3 metadata structure and validation.

This module defines the metadata format for Zarr v3 specification, which introduces a unified zarr.json metadata file format replacing the separate .zarray and .zgroup files from v2.

Key Differences from v2

  • Unified metadata: Single zarr.json file for both arrays and groups
  • Node type: Explicit node_type field ("array" or "group")
  • Codec pipeline: Unified codecs array instead of separate filters + compressor
  • Data types: Simplified type names ("float64" instead of "<f8")
  • Extensions: chunk_grid and chunk_key_encoding extension points
  • Embedded attributes: Attributes stored in metadata instead of separate .zattrs

Specification

Zarr v3 Core Specification: https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html

Examples

# Array metadata
%ExZarr.MetadataV3{
  zarr_format: 3,
  node_type: :array,
  shape: {1000, 1000},
  data_type: "float64",
  chunk_grid: %{
    name: "regular",
    configuration: %{chunk_shape: {100, 100}}
  },
  chunk_key_encoding: %{name: "default"},
  codecs: [
    %{name: "bytes"},
    %{name: "gzip", configuration: %{level: 5}}
  ],
  fill_value: 0.0,
  attributes: %{},
  dimension_names: nil
}

# Group metadata
%ExZarr.MetadataV3{
  zarr_format: 3,
  node_type: :group,
  attributes: %{"description" => "My data group"}
}

Summary

Functions

Creates v3 metadata from array configuration.

Parses JSON into MetadataV3 struct.

Converts Zarr v2 metadata to v3 format.

Extracts chunk shape from chunk_grid configuration.

Calculates the number of chunks along each dimension.

Converts metadata to JSON format.

Converts Zarr v3 metadata to v2 format (best effort).

Calculates the total number of chunks in the array.

Validates v3 metadata structure.

Types

chunk_grid()

@type chunk_grid() :: %{:name => String.t(), optional(:configuration) => map()}

chunk_key_encoding()

@type chunk_key_encoding() :: %{
  :name => String.t(),
  optional(:configuration) => map()
}

codec_spec()

@type codec_spec() :: %{:name => String.t(), optional(:configuration) => map()}

data_type()

@type data_type() :: String.t()

node_type()

@type node_type() :: :array | :group

t()

@type t() :: %ExZarr.MetadataV3{
  attributes: map(),
  chunk_grid: chunk_grid() | nil,
  chunk_key_encoding: chunk_key_encoding() | nil,
  codecs: [codec_spec()] | nil,
  data_type: data_type() | nil,
  dimension_names: [String.t() | nil] | nil,
  fill_value: term() | nil,
  node_type: node_type(),
  shape: tuple() | nil,
  zarr_format: 3
}

Functions

create(config)

@spec create(map()) :: {:ok, t()}

Creates v3 metadata from array configuration.

Converts v2-style configuration options into v3 metadata format with:

  • Unified codec pipeline (converts filters + compressor to codecs array)
  • Simplified data type names
  • Regular chunk grid
  • Default chunk key encoding

Parameters

  • config - Configuration map with keys:
    • :shape - Array dimensions (required)
    • :chunks - Chunk dimensions (required)
    • :dtype - Data type atom (required)
    • :compressor - Compressor atom (optional, default :zstd)
    • :filters - v2-style filter list (optional)
    • :codecs - v3-style codec list (takes precedence over filters/compressor)
    • :fill_value - Fill value (optional, default 0)
    • :attributes - Custom attributes (optional, default %{})

Returns

  • {:ok, metadata} - MetadataV3 struct

Examples

# Using v3 codecs directly
{:ok, metadata} = ExZarr.MetadataV3.create(%{
  shape: {1000, 1000},
  chunks: {100, 100},
  dtype: :float64,
  codecs: [
    %{name: "bytes"},
    %{name: "gzip", configuration: %{level: 5}}
  ]
})

# Using v2-style filters and compressor (auto-converted)
{:ok, metadata} = ExZarr.MetadataV3.create(%{
  shape: {1000, 1000},
  chunks: {100, 100},
  dtype: :float64,
  filters: [{:shuffle, [elementsize: 8]}],
  compressor: :zlib
})

from_json(json)

@spec from_json(String.t() | map()) :: {:ok, t()} | {:error, term()}

Parses JSON into MetadataV3 struct.

Decodes JSON (string or map) into a MetadataV3 struct, converting:

  • Lists to tuples for shape and chunk_shape
  • String node_type to atoms
  • Nested codec configurations
  • Dimension names

Parameters

  • json - JSON string or map to parse

Returns

  • {:ok, metadata} - Parsed MetadataV3 struct
  • {:error, reason} - If parsing fails

Examples

iex> json = ~S({"zarr_format":3,"node_type":"array","shape":[100],"data_type":"float64","chunk_grid":{"name":"regular","configuration":{"chunk_shape":[10]}},"chunk_key_encoding":{"name":"default"},"codecs":[{"name":"bytes"}],"fill_value":0.0,"attributes":{}})
iex> {:ok, metadata} = ExZarr.MetadataV3.from_json(json)
iex> metadata.zarr_format
3

from_v2(v2_metadata)

@spec from_v2(ExZarr.Metadata.t()) :: {:ok, t()}

Converts Zarr v2 metadata to v3 format.

Transforms v2 metadata structure to v3 with the following conversions:

  • Combines filters and compressor into unified codecs array
  • Converts NumPy dtype (e.g., "<f8") to v3 data type (e.g., "float64")
  • Creates regular chunk grid from chunks field
  • Sets default chunk key encoding
  • Preserves attributes and fill value

Parameters

  • v2_metadata - ExZarr.Metadata struct (v2 format)

Returns

  • {:ok, metadata} - MetadataV3 struct
  • {:error, reason} - If conversion fails

Examples

iex> v2_metadata = %ExZarr.Metadata{
...>   zarr_format: 2,
...>   shape: {1000, 1000},
...>   chunks: {100, 100},
...>   dtype: :float64,
...>   compressor: :zlib,
...>   fill_value: 0.0,
...>   order: "C",
...>   filters: nil
...> }
iex> {:ok, v3_metadata} = ExZarr.MetadataV3.from_v2(v2_metadata)
iex> v3_metadata.zarr_format
3
iex> v3_metadata.data_type
"float64"

Limitations

Some v2 features cannot be fully represented in v3:

  • Order ("C" vs "F") is not directly represented (row-major is default in v3)
  • Custom v2 filters may not have exact v3 codec equivalents

get_chunk_shape(arg1)

@spec get_chunk_shape(t()) :: {:ok, tuple()} | {:error, term()}

Extracts chunk shape from chunk_grid configuration.

Parameters

  • metadata - MetadataV3 struct

Returns

  • {:ok, chunk_shape} - Chunk shape tuple
  • {:error, reason} - If chunk shape cannot be extracted

Examples

iex> metadata = %ExZarr.MetadataV3{
...>   chunk_grid: %{
...>     name: "regular",
...>     configuration: %{chunk_shape: {10, 10}}
...>   }
...> }
iex> ExZarr.MetadataV3.get_chunk_shape(metadata)
{:ok, {10, 10}}

num_chunks(metadata)

@spec num_chunks(t()) :: {:ok, tuple()} | {:error, term()}

Calculates the number of chunks along each dimension.

Parameters

  • metadata - MetadataV3 struct

Returns

  • {:ok, num_chunks} - Tuple of chunk counts per dimension
  • {:error, reason} - If calculation fails

Examples

iex> metadata = %ExZarr.MetadataV3{
...>   shape: {100, 200},
...>   chunk_grid: %{
...>     name: "regular",
...>     configuration: %{chunk_shape: {10, 20}}
...>   }
...> }
iex> ExZarr.MetadataV3.num_chunks(metadata)
{:ok, {10, 10}}

to_json(metadata)

@spec to_json(t()) :: {:ok, String.t()} | {:error, term()}

Converts metadata to JSON format.

Encodes the MetadataV3 struct as JSON, converting tuples to lists and ensuring proper formatting for Zarr v3 specification compliance.

Parameters

  • metadata - MetadataV3 struct to encode

Returns

  • {:ok, json_string} - JSON encoded metadata
  • {:error, reason} - If encoding fails

Examples

iex> metadata = %ExZarr.MetadataV3{
...>   zarr_format: 3,
...>   node_type: :array,
...>   shape: {100, 200},
...>   data_type: "float64",
...>   chunk_grid: %{name: "regular", configuration: %{chunk_shape: {10, 20}}},
...>   chunk_key_encoding: %{name: "default"},
...>   codecs: [%{name: "bytes"}],
...>   fill_value: 0.0,
...>   attributes: %{}
...> }
iex> {:ok, json} = ExZarr.MetadataV3.to_json(metadata)
iex> is_binary(json)
true

to_v2(v3_metadata)

@spec to_v2(t()) :: {:ok, ExZarr.Metadata.t()} | {:error, term()}

Converts Zarr v3 metadata to v2 format (best effort).

Attempts to convert v3 metadata to v2 format with the following limitations:

  • v3-specific features (sharding, dimension names) are lost
  • Unified codec pipeline is split into filters and compressor
  • Only "regular" chunk grids are supported
  • Data type is converted from v3 to NumPy dtype

Parameters

  • v3_metadata - MetadataV3 struct

Returns

  • {:ok, metadata} - ExZarr.Metadata struct (v2 format)
  • {:error, reason} - If conversion is not possible

Examples

iex> v3_metadata = %ExZarr.MetadataV3{
...>   zarr_format: 3,
...>   node_type: :array,
...>   shape: {1000, 1000},
...>   data_type: "float64",
...>   chunk_grid: %{name: "regular", configuration: %{chunk_shape: {100, 100}}},
...>   chunk_key_encoding: %{name: "default"},
...>   codecs: [%{name: "bytes"}, %{name: "gzip", configuration: %{level: 5}}],
...>   fill_value: 0.0,
...>   attributes: %{}
...> }
iex> {:ok, v2_metadata} = ExZarr.MetadataV3.to_v2(v3_metadata)
iex> v2_metadata.zarr_format
2
iex> v2_metadata.dtype
:float64

Limitations

The following v3 features cannot be converted to v2:

  • Sharding codec (returns error)
  • Dimension names (silently dropped)
  • Irregular chunk grids (returns error)
  • Custom chunk key encodings (silently uses v2 default)
  • Array→Array codecs (transpose, quantize, bitround) may be lost

total_chunks(metadata)

@spec total_chunks(t()) :: {:ok, non_neg_integer()} | {:error, term()}

Calculates the total number of chunks in the array.

Parameters

  • metadata - MetadataV3 struct

Returns

  • {:ok, total} - Total number of chunks
  • {:error, reason} - If calculation fails

Examples

iex> metadata = %ExZarr.MetadataV3{
...>   shape: {100, 200},
...>   chunk_grid: %{
...>     name: "regular",
...>     configuration: %{chunk_shape: {10, 20}}
...>   }
...> }
iex> ExZarr.MetadataV3.total_chunks(metadata)
{:ok, 100}

validate(metadata)

@spec validate(t()) :: :ok | {:error, term()}

Validates v3 metadata structure.

Performs comprehensive validation including:

  • Zarr format version check
  • Node type validation
  • Array-specific field requirements
  • Codec pipeline validation
  • Extension format checks

Parameters

  • metadata - MetadataV3 struct to validate

Returns

  • :ok if metadata is valid
  • {:error, reason} if validation fails

Examples

iex> metadata = %ExZarr.MetadataV3{
...>   zarr_format: 3,
...>   node_type: :array,
...>   shape: {100},
...>   data_type: "float64",
...>   chunk_grid: %{name: "regular", configuration: %{chunk_shape: {10}}},
...>   chunk_key_encoding: %{name: "default"},
...>   codecs: [%{name: "bytes"}],
...>   fill_value: 0.0,
...>   attributes: %{}
...> }
iex> ExZarr.MetadataV3.validate(metadata)
:ok