ExDataSketch.Storage.DETS (ExDataSketch v0.9.0)

Copy Markdown View Source

DETS-backed persistence for sketches.

Stores serialized EXSK v2 frames in a DETS table. DETS provides disk-backed storage that survives process and node restarts, at the cost of slower operations compared to ETS.

Table Requirements

  • The DETS table must be :set type. :ordered_set is not supported by DETS.
  • The table must be opened by the caller before use.
  • DETS has a 2GB file size limit.

Concurrency

  • DETS uses file-level locking. Concurrent writes are serialized.
  • merge/3 performs a read-modify-write cycle while holding the table lock.

Examples

# Open a DETS table (application concern) {:ok, _} = :dets.open_file(:sketches, [type: :set])

# Save a sketch :ok = ExDataSketch.Storage.DETS.save(sketch, :sketches, "cardinality:2024-01")

# Load a sketch {:ok, sketch} = ExDataSketch.Storage.DETS.load(ExDataSketch.HLL, :sketches, "cardinality:2024-01")

# Close when done :ok = :dets.close(:sketches)

Summary

Functions

delete(table, key)

Deletes a sketch from the DETS table by key.

load(sketch_module, table, key)

Loads a sketch from the DETS table by key.

merge(sketch, table, key)

Merges a sketch into the persisted value at the given key.

save(sketch, table, key)

Persists a sketch under the given key in the DETS table.

Functions

delete(table, key)

@spec delete(atom(), ExDataSketch.Storage.key()) :: :ok | {:error, term()}

Deletes a sketch from the DETS table by key.

Arguments

  • table -- the DETS table name (atom).
  • key -- the key to delete.

Returns

  • :ok on success (including when the key did not exist).
  • {:error, reason} if the DETS operation fails.

Examples

iex> :dets.open_file(:test_dets_del, type: :set)
iex> sketch = ExDataSketch.HLL.new(p: 10)
iex> ExDataSketch.Storage.DETS.save(sketch, :test_dets_del, "hll:test")
:ok
iex> ExDataSketch.Storage.DETS.delete(:test_dets_del, "hll:test")
:ok
iex> ExDataSketch.Storage.DETS.load(ExDataSketch.HLL, :test_dets_del, "hll:test")
{:error, :not_found}
iex> :dets.close(:test_dets_del)
:ok

load(sketch_module, table, key)

@spec load(module(), atom(), ExDataSketch.Storage.key()) ::
  {:ok, struct()} | {:error, :not_found | term()}

Loads a sketch from the DETS table by key.

The binary value is deserialized using the given sketch module's deserialize/1 function.

Arguments

  • sketch_module -- the sketch module atom (e.g., ExDataSketch.HLL).
  • table -- the DETS table name (atom).
  • key -- the key to look up.

Returns

  • {:ok, sketch} on success.
  • {:error, :not_found} if the key does not exist.
  • {:error, %DeserializationError{}} if the stored binary is corrupted.
  • {:error, reason} on DETS or other deserialization failures.

Examples

iex> :dets.open_file(:test_dets_load, type: :set)
iex> sketch = ExDataSketch.HLL.new(p: 10) |> ExDataSketch.HLL.update("a")
iex> ExDataSketch.Storage.DETS.save(sketch, :test_dets_load, "hll:test")
:ok
iex> {:ok, loaded} = ExDataSketch.Storage.DETS.load(ExDataSketch.HLL, :test_dets_load, "hll:test")
iex> ExDataSketch.HLL.estimate(loaded) > 0.0
true
iex> ExDataSketch.Storage.DETS.load(ExDataSketch.HLL, :test_dets_load, "nonexistent")
{:error, :not_found}
iex> :dets.close(:test_dets_load)
:ok

merge(sketch, table, key)

@spec merge(struct(), atom(), ExDataSketch.Storage.key()) :: :ok | {:error, term()}

Merges a sketch into the persisted value at the given key.

If no sketch exists at the key, this is equivalent to save/3. Otherwise, the persisted sketch is loaded, merged with the given sketch, and saved back.

The read-modify-write cycle occurs while the DETS table lock is held. This provides atomicity for single-node writers. DETS file-level locking does not extend across distributed nodes; for distributed atomicity, use ExDataSketch.Storage.Mnesia.

Arguments

  • sketch -- the sketch to merge into the persisted value.
  • table -- the DETS table name (atom).
  • key -- the key whose persisted sketch to merge into.

Returns

:ok on success, or {:error, reason} on failure.

Examples

iex> :dets.open_file(:test_dets_merge, type: :set)
iex> sketch_a = ExDataSketch.HLL.new(p: 10) |> ExDataSketch.HLL.update("a")
iex> ExDataSketch.Storage.DETS.save(sketch_a, :test_dets_merge, "hll:test")
:ok
iex> sketch_b = ExDataSketch.HLL.new(p: 10) |> ExDataSketch.HLL.update("b")
iex> ExDataSketch.Storage.DETS.merge(sketch_b, :test_dets_merge, "hll:test")
:ok
iex> {:ok, merged} = ExDataSketch.Storage.DETS.load(ExDataSketch.HLL, :test_dets_merge, "hll:test")
iex> ExDataSketch.HLL.estimate(merged) >= 1.9
true
iex> :dets.close(:test_dets_merge)
:ok

save(sketch, table, key)

@spec save(struct(), atom(), ExDataSketch.Storage.key()) :: :ok | {:error, term()}

Persists a sketch under the given key in the DETS table.

The sketch is serialized to an EXSK v2 binary frame before storage.

Arguments

  • sketch -- a sketch struct.
  • table -- the DETS table name (atom).
  • key -- the key under which to store the sketch.

Returns

:ok on success, or {:error, reason} on failure.

Examples

iex> :dets.open_file(:test_dets_save, type: :set)
iex> sketch = ExDataSketch.HLL.new(p: 10) |> ExDataSketch.HLL.update("a")
iex> ExDataSketch.Storage.DETS.save(sketch, :test_dets_save, "hll:test")
:ok
iex> :dets.close(:test_dets_save)
:ok