ExZarr.Storage.Backend.ETS (ExZarr v1.1.0)

View Source

ETS (Erlang Term Storage) storage backend for Zarr arrays.

Stores chunks and metadata in ETS tables, providing fast in-memory storage with multi-process access capabilities. Unlike the Memory backend which uses an Agent, ETS tables are shared memory accessible by multiple processes.

Configuration

Optional configuration:

  • :table_name - ETS table name (optional, auto-generated if not provided)
  • :named_table - Use named table (optional, default: true)
  • :table_type - Table type (optional, default: :set)
  • :access - Access level (optional, default: :public)
  • :heir - Heir process for table (optional, default: :none)

Dependencies

No external dependencies - ETS is built into Erlang/OTP.

Example

# Register the ETS backend
:ok = ExZarr.Storage.Registry.register(ExZarr.Storage.Backend.ETS)

# Create array with ETS storage
{:ok, array} = ExZarr.create(
  shape: {1000, 1000},
  chunks: {100, 100},
  dtype: :float64,
  storage: :ets,
  table_name: :my_array_data
)

# Write and read data
ExZarr.Array.set_slice(array, data, start: {0, 0}, stop: {100, 100})
{:ok, result} = ExZarr.Array.get_slice(array, start: {0, 0}, stop: {100, 100})

Table Types

  • :set - One value per unique key (default)
  • :ordered_set - Sorted set (slower writes, faster ordered access)
  • :bag - Multiple values per key allowed
  • :duplicate_bag - Multiple identical values per key allowed

Access Levels

  • :public - Any process can read/write (default)
  • :protected - Any process can read, only owner can write
  • :private - Only owner process can read/write

Named Tables

Named tables can be accessed by name from any process:

# Create with named table
{:ok, array} = ExZarr.create(
  storage: :ets,
  table_name: :shared_array,
  named_table: true,
  access: :public
)

# Access from different process
:ets.lookup(:shared_array, {:chunk, {0}})

Table Ownership

ETS tables are owned by the creating process by default. When that process dies, the table is deleted. Use the :heir option to transfer ownership:

{:ok, array} = ExZarr.create(
  storage: :ets,
  heir: self()
)

Performance Characteristics

  • Very fast: Direct memory access, no message passing
  • Concurrent: Multiple readers, single writer
  • Scalable: Efficient for large datasets
  • Non-persistent: Data lost on table deletion

Use Cases

  • Multi-process data sharing
  • High-performance caching
  • Concurrent read workloads
  • Inter-process communication
  • When you need faster access than Agent/GenServer

Comparison with Memory Backend

FeatureMemory (Agent)ETS
AccessMessage passingDirect memory
ConcurrencySequentialConcurrent reads
SpeedFastVery fast
SharingVia messagesDirect access
Best forSingle processMulti-process

Error Handling

ETS errors are returned as {:error, reason} tuples. Common errors:

  • :table_not_found - Table doesn't exist
  • :not_found - Key doesn't exist
  • :access_denied - Permission error

Summary

Functions

Clear all data from an ETS table without deleting it.

Delete an ETS table.

Get the current state (all chunks and metadata) from an ETS table.

Check if an ETS table exists.

Get information about an ETS table.

Functions

clear_table(table)

@spec clear_table(atom() | :ets.tid()) :: :ok | {:error, :table_not_found}

Clear all data from an ETS table without deleting it.

Example

:ok = ExZarr.Storage.Backend.ETS.clear_table(:my_array_data)

delete_table(table)

@spec delete_table(atom() | :ets.tid()) :: :ok | {:error, :table_not_found}

Delete an ETS table.

Useful for cleanup in tests or when you want to explicitly remove a table.

Example

:ok = ExZarr.Storage.Backend.ETS.delete_table(:my_array_data)

get_state(table)

@spec get_state(atom() | :ets.tid()) :: %{chunks: map(), metadata: binary() | nil}

Get the current state (all chunks and metadata) from an ETS table.

Useful for debugging and testing.

Example

state = ExZarr.Storage.Backend.ETS.get_state(:my_array_data)
assert map_size(state.chunks) == 5

table_exists?(table)

@spec table_exists?(atom() | :ets.tid()) :: boolean()

Check if an ETS table exists.

Example

if ExZarr.Storage.Backend.ETS.table_exists?(:my_array_data) do
  IO.puts("Table exists")
end

table_info(table)

@spec table_info(atom() | :ets.tid()) :: map() | {:error, :table_not_found}

Get information about an ETS table.

Example

info = ExZarr.Storage.Backend.ETS.table_info(:my_array_data)
IO.inspect(info.size)