ExZarr.ChunkKey (ExZarr v1.1.0)

View Source

Chunk key encoding for Zarr v2 and v3 formats.

Zarr uses chunk keys to identify individual chunk files in storage. The format differs between v2 and v3 specifications:

Zarr v2 Format

Dot-separated notation with optional dimension separator:

  • 1D: "0", "1", "2"
  • 2D: "0.0", "0.1", "1.0"
  • 3D: "0.0.0", "0.1.2", "1.2.3"

Zarr v3 Format

Slash-separated notation with c/ prefix:

  • 1D: "c/0", "c/1", "c/2"
  • 2D: "c/0/0", "c/0/1", "c/1/0"
  • 3D: "c/0/0/0", "c/0/1/2", "c/1/2/3"

Usage

# Encode chunk indices to keys
iex> ExZarr.ChunkKey.encode({0, 1, 2}, 2)
"0.1.2"

iex> ExZarr.ChunkKey.encode({0, 1, 2}, 3)
"c/0/1/2"

# Decode keys to indices
iex> ExZarr.ChunkKey.decode("0.1.2", 2)
{:ok, {0, 1, 2}}

iex> ExZarr.ChunkKey.decode("c/0/1/2", 3)
{:ok, {0, 1, 2}}

Storage Backends

Storage backends must use this module to ensure correct chunk key format for the array version being used.

Summary

Types

chunk_index()
chunk_key()
version()

Functions

chunk_directory(base_path, int)

Builds a chunk directory path for v3 format.

chunk_key_pattern(int)

Returns a regex pattern for matching valid chunk keys.

chunk_path(base_path, chunk_index, version)

Builds the full path to a chunk file.

decode(chunk_key, int)

Decodes a chunk key string into a chunk index tuple.

decode_with(chunk_key, encoder_name, opts \\ [])

Decodes a chunk key using a named encoder.

encode(chunk_index, int)

Encodes a chunk index tuple into a chunk key string.

encode_with(chunk_index, encoder_name, opts \\ [])

Encodes a chunk index using a named encoder.

list_all(shape, chunks, version)

Lists all possible chunk keys for an array based on its shape and chunk size.

register_encoder(name, encoder_module)

Registers a custom chunk key encoder.

valid?(chunk_key, version)

Validates that a chunk key matches the expected format for a version.

Types

chunk_index()

@type chunk_index() :: tuple()

chunk_key()

@type chunk_key() :: String.t()

version()

@type version() :: 2 | 3

Functions

chunk_directory(base_path, int)

@spec chunk_directory(String.t(), version()) :: String.t()

Builds a chunk directory path for v3 format.

In v3, chunks are stored in a c/ subdirectory. This function returns the directory path where chunks should be stored.

Parameters

  • base_path - Base array path
  • version - Zarr format version (2 or 3)

Returns

  • Directory path for chunks

Examples

iex> ExZarr.ChunkKey.chunk_directory("/data/array", 2)
"/data/array"

iex> ExZarr.ChunkKey.chunk_directory("/data/array", 3)
"/data/array/c"

chunk_key_pattern(int)

@spec chunk_key_pattern(version()) :: Regex.t()

Returns a regex pattern for matching valid chunk keys.

Parameters

  • version - Zarr format version (2 or 3)

Returns

  • Regex pattern for valid chunk keys

Examples

iex> pattern = ExZarr.ChunkKey.chunk_key_pattern(2)
iex> Regex.match?(pattern, "0.1.2")
true

iex> pattern = ExZarr.ChunkKey.chunk_key_pattern(3)
iex> Regex.match?(pattern, "c/0/1/2")
true

chunk_path(base_path, chunk_index, version)

@spec chunk_path(String.t(), chunk_index(), version()) :: String.t()

Builds the full path to a chunk file.

Parameters

  • base_path - Base array path
  • chunk_index - Chunk index tuple
  • version - Zarr format version (2 or 3)

Returns

  • Full path to chunk file

Examples

iex> ExZarr.ChunkKey.chunk_path("/data/array", {0, 1}, 2)
"/data/array/0.1"

iex> ExZarr.ChunkKey.chunk_path("/data/array", {0, 1}, 3)
"/data/array/c/0/1"

decode(chunk_key, int)

@spec decode(chunk_key(), version()) ::
  {:ok, chunk_index()} | {:error, :invalid_chunk_key}

Decodes a chunk key string into a chunk index tuple.

Parameters

  • chunk_key - Chunk key string
  • version - Zarr format version (2 or 3)

Returns

  • {:ok, chunk_index} - Tuple of integers
  • {:error, :invalid_chunk_key} - If key format is invalid

Examples

# v2 format
iex> ExZarr.ChunkKey.decode("0", 2)
{:ok, {0}}

iex> ExZarr.ChunkKey.decode("0.1.2", 2)
{:ok, {0, 1, 2}}

# v3 format
iex> ExZarr.ChunkKey.decode("c/0", 3)
{:ok, {0}}

iex> ExZarr.ChunkKey.decode("c/0/1/2", 3)
{:ok, {0, 1, 2}}

# Invalid keys
iex> ExZarr.ChunkKey.decode("invalid", 2)
{:error, :invalid_chunk_key}

iex> ExZarr.ChunkKey.decode("0.1.2", 3)
{:error, :invalid_chunk_key}

decode_with(chunk_key, encoder_name, opts \\ [])

@spec decode_with(chunk_key(), atom() | version(), keyword()) ::
  {:ok, chunk_index()} | {:error, term()}

Decodes a chunk key using a named encoder.

Falls back to version-based decoding if encoder not found.

Parameters

  • chunk_key - String representation

  • encoder_name - Atom identifying the encoder, or version number (2 | 3)

  • opts - Options to pass to encoder

Examples

ExZarr.ChunkKey.decode_with("chunk_0_1_2", :custom, [])
{:ok, {0, 1, 2}}

ExZarr.ChunkKey.decode_with("0.1", :v2, [])
{:ok, {0, 1}}

encode(chunk_index, int)

@spec encode(chunk_index(), version()) :: chunk_key()

Encodes a chunk index tuple into a chunk key string.

Parameters

  • chunk_index - Tuple of non-negative integers representing chunk coordinates
  • version - Zarr format version (2 or 3)

Returns

  • Chunk key string in the appropriate format

Examples

# v2 format
iex> ExZarr.ChunkKey.encode({0}, 2)
"0"

iex> ExZarr.ChunkKey.encode({0, 1}, 2)
"0.1"

iex> ExZarr.ChunkKey.encode({42}, 2)
"42"

# v3 format
iex> ExZarr.ChunkKey.encode({0}, 3)
"c/0"

iex> ExZarr.ChunkKey.encode({0, 1, 2}, 3)
"c/0/1/2"

encode_with(chunk_index, encoder_name, opts \\ [])

@spec encode_with(chunk_index(), atom() | version(), keyword()) :: chunk_key()

Encodes a chunk index using a named encoder.

Falls back to version-based encoding if encoder not found.

Parameters

  • chunk_index - Tuple of integers

  • encoder_name - Atom identifying the encoder, or version number (2 | 3)

  • opts - Options to pass to encoder

Examples

ExZarr.ChunkKey.encode_with({0, 1, 2}, :custom, [])
"chunk_0_1_2"

ExZarr.ChunkKey.encode_with({0, 1}, :v2, [])
"0.1"

list_all(shape, chunks, version)

@spec list_all(tuple(), tuple(), version()) :: [chunk_key()]

Lists all possible chunk keys for an array based on its shape and chunk size.

Parameters

  • shape - Array shape tuple
  • chunks - Chunk size tuple
  • version - Zarr format version (2 or 3)

Returns

  • List of all chunk key strings

Examples

iex> ExZarr.ChunkKey.list_all({10}, {5}, 2)
["0", "1"]

iex> ExZarr.ChunkKey.list_all({10, 10}, {5, 5}, 3)
["c/0/0", "c/0/1", "c/1/0", "c/1/1"]

register_encoder(name, encoder_module)

@spec register_encoder(atom(), module()) :: :ok

Registers a custom chunk key encoder.

Parameters

Examples

defmodule MyEncoder do
  @behaviour ExZarr.ChunkKey.Encoder
  # ... implement callbacks ...
end

ExZarr.ChunkKey.register_encoder(:my_format, MyEncoder)

valid?(chunk_key, version)

@spec valid?(chunk_key(), version()) :: boolean()

Validates that a chunk key matches the expected format for a version.

Parameters

  • chunk_key - Chunk key string to validate
  • version - Zarr format version (2 or 3)

Returns

  • true if chunk key is valid for the version
  • false otherwise

Examples

iex> ExZarr.ChunkKey.valid?("0.1.2", 2)
true

iex> ExZarr.ChunkKey.valid?("c/0/1/2", 2)
false

iex> ExZarr.ChunkKey.valid?("c/0/1/2", 3)
true

iex> ExZarr.ChunkKey.valid?("0.1.2", 3)
false