ExZarr.MetadataV3 (ExZarr v1.1.0)
View SourceZarr v3 metadata structure and validation.
This module defines the metadata format for Zarr v3 specification, which
introduces a unified zarr.json metadata file format replacing the separate
.zarray and .zgroup files from v2.
Key Differences from v2
- Unified metadata: Single
zarr.jsonfile for both arrays and groups - Node type: Explicit
node_typefield ("array" or "group") - Codec pipeline: Unified
codecsarray instead of separatefilters+compressor - Data types: Simplified type names ("float64" instead of "<f8")
- Extensions:
chunk_gridandchunk_key_encodingextension points - Embedded attributes: Attributes stored in metadata instead of separate
.zattrs
Specification
Zarr v3 Core Specification: https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html
Examples
# Array metadata
%ExZarr.MetadataV3{
zarr_format: 3,
node_type: :array,
shape: {1000, 1000},
data_type: "float64",
chunk_grid: %{
name: "regular",
configuration: %{chunk_shape: {100, 100}}
},
chunk_key_encoding: %{name: "default"},
codecs: [
%{name: "bytes"},
%{name: "gzip", configuration: %{level: 5}}
],
fill_value: 0.0,
attributes: %{},
dimension_names: nil
}
# Group metadata
%ExZarr.MetadataV3{
zarr_format: 3,
node_type: :group,
attributes: %{"description" => "My data group"}
}
Summary
Functions
Creates v3 metadata from array configuration.
Parses JSON into MetadataV3 struct.
Converts Zarr v2 metadata to v3 format.
Extracts chunk shape from chunk_grid configuration.
Calculates the number of chunks along each dimension.
Converts metadata to JSON format.
Converts Zarr v3 metadata to v2 format (best effort).
Calculates the total number of chunks in the array.
Validates v3 metadata structure.
Types
@type data_type() :: String.t()
@type node_type() :: :array | :group
@type t() :: %ExZarr.MetadataV3{ attributes: map(), chunk_grid: chunk_grid() | nil, chunk_key_encoding: chunk_key_encoding() | nil, codecs: [codec_spec()] | nil, data_type: data_type() | nil, dimension_names: [String.t() | nil] | nil, fill_value: term() | nil, node_type: node_type(), shape: tuple() | nil, zarr_format: 3 }
Functions
Creates v3 metadata from array configuration.
Converts v2-style configuration options into v3 metadata format with:
- Unified codec pipeline (converts filters + compressor to codecs array)
- Simplified data type names
- Regular chunk grid
- Default chunk key encoding
Parameters
config- Configuration map with keys::shape- Array dimensions (required):chunks- Chunk dimensions (required):dtype- Data type atom (required):compressor- Compressor atom (optional, default :zstd):filters- v2-style filter list (optional):codecs- v3-style codec list (takes precedence over filters/compressor):fill_value- Fill value (optional, default 0):attributes- Custom attributes (optional, default %{})
Returns
{:ok, metadata}- MetadataV3 struct
Examples
# Using v3 codecs directly
{:ok, metadata} = ExZarr.MetadataV3.create(%{
shape: {1000, 1000},
chunks: {100, 100},
dtype: :float64,
codecs: [
%{name: "bytes"},
%{name: "gzip", configuration: %{level: 5}}
]
})
# Using v2-style filters and compressor (auto-converted)
{:ok, metadata} = ExZarr.MetadataV3.create(%{
shape: {1000, 1000},
chunks: {100, 100},
dtype: :float64,
filters: [{:shuffle, [elementsize: 8]}],
compressor: :zlib
})
Parses JSON into MetadataV3 struct.
Decodes JSON (string or map) into a MetadataV3 struct, converting:
- Lists to tuples for shape and chunk_shape
- String node_type to atoms
- Nested codec configurations
- Dimension names
Parameters
json- JSON string or map to parse
Returns
{:ok, metadata}- Parsed MetadataV3 struct{:error, reason}- If parsing fails
Examples
iex> json = ~S({"zarr_format":3,"node_type":"array","shape":[100],"data_type":"float64","chunk_grid":{"name":"regular","configuration":{"chunk_shape":[10]}},"chunk_key_encoding":{"name":"default"},"codecs":[{"name":"bytes"}],"fill_value":0.0,"attributes":{}})
iex> {:ok, metadata} = ExZarr.MetadataV3.from_json(json)
iex> metadata.zarr_format
3
@spec from_v2(ExZarr.Metadata.t()) :: {:ok, t()}
Converts Zarr v2 metadata to v3 format.
Transforms v2 metadata structure to v3 with the following conversions:
- Combines
filtersandcompressorinto unifiedcodecsarray - Converts NumPy dtype (e.g., "<f8") to v3 data type (e.g., "float64")
- Creates regular chunk grid from
chunksfield - Sets default chunk key encoding
- Preserves attributes and fill value
Parameters
v2_metadata- ExZarr.Metadata struct (v2 format)
Returns
{:ok, metadata}- MetadataV3 struct{:error, reason}- If conversion fails
Examples
iex> v2_metadata = %ExZarr.Metadata{
...> zarr_format: 2,
...> shape: {1000, 1000},
...> chunks: {100, 100},
...> dtype: :float64,
...> compressor: :zlib,
...> fill_value: 0.0,
...> order: "C",
...> filters: nil
...> }
iex> {:ok, v3_metadata} = ExZarr.MetadataV3.from_v2(v2_metadata)
iex> v3_metadata.zarr_format
3
iex> v3_metadata.data_type
"float64"Limitations
Some v2 features cannot be fully represented in v3:
- Order ("C" vs "F") is not directly represented (row-major is default in v3)
- Custom v2 filters may not have exact v3 codec equivalents
Extracts chunk shape from chunk_grid configuration.
Parameters
metadata- MetadataV3 struct
Returns
{:ok, chunk_shape}- Chunk shape tuple{:error, reason}- If chunk shape cannot be extracted
Examples
iex> metadata = %ExZarr.MetadataV3{
...> chunk_grid: %{
...> name: "regular",
...> configuration: %{chunk_shape: {10, 10}}
...> }
...> }
iex> ExZarr.MetadataV3.get_chunk_shape(metadata)
{:ok, {10, 10}}
Calculates the number of chunks along each dimension.
Parameters
metadata- MetadataV3 struct
Returns
{:ok, num_chunks}- Tuple of chunk counts per dimension{:error, reason}- If calculation fails
Examples
iex> metadata = %ExZarr.MetadataV3{
...> shape: {100, 200},
...> chunk_grid: %{
...> name: "regular",
...> configuration: %{chunk_shape: {10, 20}}
...> }
...> }
iex> ExZarr.MetadataV3.num_chunks(metadata)
{:ok, {10, 10}}
Converts metadata to JSON format.
Encodes the MetadataV3 struct as JSON, converting tuples to lists and ensuring proper formatting for Zarr v3 specification compliance.
Parameters
metadata- MetadataV3 struct to encode
Returns
{:ok, json_string}- JSON encoded metadata{:error, reason}- If encoding fails
Examples
iex> metadata = %ExZarr.MetadataV3{
...> zarr_format: 3,
...> node_type: :array,
...> shape: {100, 200},
...> data_type: "float64",
...> chunk_grid: %{name: "regular", configuration: %{chunk_shape: {10, 20}}},
...> chunk_key_encoding: %{name: "default"},
...> codecs: [%{name: "bytes"}],
...> fill_value: 0.0,
...> attributes: %{}
...> }
iex> {:ok, json} = ExZarr.MetadataV3.to_json(metadata)
iex> is_binary(json)
true
@spec to_v2(t()) :: {:ok, ExZarr.Metadata.t()} | {:error, term()}
Converts Zarr v3 metadata to v2 format (best effort).
Attempts to convert v3 metadata to v2 format with the following limitations:
- v3-specific features (sharding, dimension names) are lost
- Unified codec pipeline is split into filters and compressor
- Only "regular" chunk grids are supported
- Data type is converted from v3 to NumPy dtype
Parameters
v3_metadata- MetadataV3 struct
Returns
{:ok, metadata}- ExZarr.Metadata struct (v2 format){:error, reason}- If conversion is not possible
Examples
iex> v3_metadata = %ExZarr.MetadataV3{
...> zarr_format: 3,
...> node_type: :array,
...> shape: {1000, 1000},
...> data_type: "float64",
...> chunk_grid: %{name: "regular", configuration: %{chunk_shape: {100, 100}}},
...> chunk_key_encoding: %{name: "default"},
...> codecs: [%{name: "bytes"}, %{name: "gzip", configuration: %{level: 5}}],
...> fill_value: 0.0,
...> attributes: %{}
...> }
iex> {:ok, v2_metadata} = ExZarr.MetadataV3.to_v2(v3_metadata)
iex> v2_metadata.zarr_format
2
iex> v2_metadata.dtype
:float64Limitations
The following v3 features cannot be converted to v2:
- Sharding codec (returns error)
- Dimension names (silently dropped)
- Irregular chunk grids (returns error)
- Custom chunk key encodings (silently uses v2 default)
- Array→Array codecs (transpose, quantize, bitround) may be lost
@spec total_chunks(t()) :: {:ok, non_neg_integer()} | {:error, term()}
Calculates the total number of chunks in the array.
Parameters
metadata- MetadataV3 struct
Returns
{:ok, total}- Total number of chunks{:error, reason}- If calculation fails
Examples
iex> metadata = %ExZarr.MetadataV3{
...> shape: {100, 200},
...> chunk_grid: %{
...> name: "regular",
...> configuration: %{chunk_shape: {10, 20}}
...> }
...> }
iex> ExZarr.MetadataV3.total_chunks(metadata)
{:ok, 100}
Validates v3 metadata structure.
Performs comprehensive validation including:
- Zarr format version check
- Node type validation
- Array-specific field requirements
- Codec pipeline validation
- Extension format checks
Parameters
metadata- MetadataV3 struct to validate
Returns
:okif metadata is valid{:error, reason}if validation fails
Examples
iex> metadata = %ExZarr.MetadataV3{
...> zarr_format: 3,
...> node_type: :array,
...> shape: {100},
...> data_type: "float64",
...> chunk_grid: %{name: "regular", configuration: %{chunk_shape: {10}}},
...> chunk_key_encoding: %{name: "default"},
...> codecs: [%{name: "bytes"}],
...> fill_value: 0.0,
...> attributes: %{}
...> }
iex> ExZarr.MetadataV3.validate(metadata)
:ok