ExZarr.Storage.Backend.Mnesia (ExZarr v1.1.0)

View Source

Mnesia distributed database storage backend for Zarr arrays.

Stores chunks and metadata in Mnesia, Erlang's built-in distributed database. Provides ACID transactions, replication, and fault tolerance.

Configuration

Requires the following options:

  • :table_name - Mnesia table name for storing array data (optional, default: :zarr_storage)
  • :array_id - Unique identifier for this array (required)
  • :node - Mnesia node (optional, default: current node)
  • :ram_copies - Use RAM copies instead of disc copies (optional, default: false)

Dependencies

No external dependencies - Mnesia is built into Erlang/OTP.

Setup

Mnesia must be initialized before use:

# In your application startup
mnesia().create_schema([node()])
mnesia().start()

Example

# Register the Mnesia backend
:ok = ExZarr.Storage.Registry.register(ExZarr.Storage.Backend.Mnesia)

# Create array with Mnesia storage
{:ok, array} = ExZarr.create(
  shape: {1000, 1000},
  chunks: {100, 100},
  dtype: :float64,
  storage: :mnesia,
  array_id: "experiment_001"
)

# Write and read data
ExZarr.Array.set_slice(array, data, start: {0, 0}, stop: {100, 100})
{:ok, result} = ExZarr.Array.get_slice(array, start: {0, 0}, stop: {100, 100})

Mnesia Table Structure

Data is stored in a table with the following schema:

{:zarr_storage, {array_id, chunk_index}, data}
{:zarr_storage, {array_id, :metadata}, metadata_json}

Performance Considerations

  • RAM tables provide fastest access but limited by memory
  • Disc tables persist across restarts
  • Distributed tables can be replicated across nodes
  • Consider fragmentation for very large tables

Distributed Operation

Mnesia supports multi-node operation:

# Create distributed table
{:ok, array} = ExZarr.create(
  storage: :mnesia,
  array_id: "shared_array",
  nodes: [node(), :"node2@host", :"node3@host"]
)

Error Handling

Mnesia errors are returned as {:error, reason} tuples. Common errors:

  • :table_not_found - Table doesn't exist
  • :not_found - Record doesn't exist
  • :transaction_error - Transaction failed