Persistence Snapshots

Copy Markdown View Source
Run in Livebook 
Mix.install([
  {:ex_data_sketch, "~> 0.10"}
])

Introduction

ExDataSketch provides five persistence backends for saving, loading, and merging sketches. All backends serialize sketches using the EXSK v2 binary format with CRC32C checksums, ensuring data integrity across storage boundaries.

This Livebook demonstrates ETS (in-memory) and DETS (disk-backed) storage, plus serialization for network transfer.

Section 1: ETS -- Fast In-Memory Storage

ETS provides the fastest persistence option. Data lives as long as the owning process. Ideal for caching, process-local state, and read-heavy workloads.

# Create an ETS table
table = :ets.new(:sketch_cache, [:set, :public])

# Build a sketch and save it
sketch = ExDataSketch.HLL.from_enumerable(1..10_000, p: 14)
:ok = ExDataSketch.Storage.ETS.save(sketch, table, "daily:2024-01-15")

# Load it back
{:ok, loaded} = ExDataSketch.Storage.ETS.load(ExDataSketch.HLL, table, "daily:2024-01-15")
IO.puts("Loaded cardinality: #{Float.round(ExDataSketch.HLL.estimate(loaded), 0)} (true: 10000)")

# Atomic merge: merge a partial sketch into the stored value
partial = ExDataSketch.HLL.from_enumerable(10_001..15_000, p: 14)
:ok = ExDataSketch.Storage.ETS.merge(partial, table, "daily:2024-01-15")

# Verify the merge
{:ok, merged} = ExDataSketch.Storage.ETS.load(ExDataSketch.HLL, table, "daily:2024-01-15")
IO.puts("After merge: #{Float.round(ExDataSketch.HLL.estimate(merged), 0)} (expected ~15000)")

# Cleanup
:ok = ExDataSketch.Storage.ETS.delete(table, "daily:2024-01-15")
:ets.delete(table)

Section 2: DETS -- Durable Disk Storage

DETS stores sketches on disk, surviving process and node restarts. Slower than ETS but provides durability without requiring a database.

# Open a DETS table
{:ok, _} = :dets.open_file(:sketch_dets, [type: :set])

# Save a sketch
sketch = ExDataSketch.ULL.from_enumerable(1..5000, p: 14)
:ok = ExDataSketch.Storage.DETS.save(sketch, :sketch_dets, "metrics:hourly")

# Load it back
{:ok, loaded} = ExDataSketch.Storage.DETS.load(ExDataSketch.ULL, :sketch_dets, "metrics:hourly")
IO.puts("Loaded ULL cardinality: #{Float.round(ExDataSketch.ULL.estimate(loaded), 0)} (true: 5000)")

# Merge into stored value
partial = ExDataSketch.ULL.from_enumerable(5001..8000, p: 14)
:ok = ExDataSketch.Storage.DETS.merge(partial, :sketch_dets, "metrics:hourly")

{:ok, merged} = ExDataSketch.Storage.DETS.load(ExDataSketch.ULL, :sketch_dets, "metrics:hourly")
IO.puts("After merge: #{Float.round(ExDataSketch.ULL.estimate(merged), 0)} (expected ~8000)")

# Cleanup
:ok = ExDataSketch.Storage.DETS.delete(:sketch_dets, "metrics:hourly")
:ok = :dets.close(:sketch_dets)

Section 3: Serialization and Transport

Serialize sketches for network transfer, message queues, or file storage:

sketch = ExDataSketch.CMS.from_enumerable(
  Stream.map(1..10_000, fn i -> "item_#{rem(i, 100)}" end),
  width: 128,
  depth: 5
)

# Serialize to binary (EXSK v2 with CRC32C checksum)
binary = ExDataSketch.CMS.serialize(sketch)
IO.puts("Serialized size: #{byte_size(binary)} bytes")

# Deserialize
{:ok, restored} = ExDataSketch.CMS.deserialize(binary)
IO.puts("Count of 'item_1': #{ExDataSketch.CMS.estimate(restored, "item_1")} (true: 100)")

# Performance comparison: serialize overhead
sketch_h = ExDataSketch.HLL.from_enumerable(1..10_000, p: 14)
{ser_time, binary} = :timer.tc(fn -> ExDataSketch.HLL.serialize(sketch_h) end)
IO.puts("HLL serialize: #{div(ser_time, 1000)}ms for #{byte_size(binary)} bytes")

# Deserialize round-trip
{:ok, roundtrip} = ExDataSketch.HLL.deserialize(binary)
IO.puts("Round-trip estimate: #{Float.round(ExDataSketch.HLL.estimate(roundtrip), 0)} (true: 10000, original: #{Float.round(ExDataSketch.HLL.estimate(sketch_h), 0)})")

Section 4: Multi-Backend Strategy

# Pattern: ETS for hot data, DETS for warm, serialize for cold storage
hot_table = :ets.new(:hot_sketches, [:set, :public])

# Hot: in-memory ETS for sub-millisecond access
sketch = ExDataSketch.HLL.from_enumerable(1..50_000, p: 14)
:ok = ExDataSketch.Storage.ETS.save(sketch, hot_table, "active:users")

# Warm: periodic DETS snapshots for restart recovery
{:ok, _} = :dets.open_file(:warm_sketches, [type: :set])
:ok = ExDataSketch.Storage.DETS.save(sketch, :warm_sketches, "active:users")

# Cold: serialized binary for archival or cross-node transfer
binary = ExDataSketch.HLL.serialize(sketch)
IO.puts("Archive size: #{byte_size(binary)} bytes")

# Cleanup
:ets.delete(hot_table)
:ok = :dets.close(:warm_sketches)

Section 5: Operational Guidance

Choose ETS when:

  • You need sub-millisecond reads
  • Data can be lost on node restart (ephemeral caching)
  • Multiple processes need concurrent read access

Choose DETS when:

  • You need durability across process restarts
  • Write throughput is moderate (DETS serializes writes)
  • 2GB file size limit is acceptable

Choose external storage (Ecto, Mnesia, CubDB) when:

  • You need multi-node replication (Mnesia)
  • You need query capabilities (Ecto/SQL)
  • You need atomic multi-key transactions (CubDB)

Serialization overhead:

  • HLL at p=14: 16KB serialized (negligible serialize/deserialize cost)
  • CMS (128x5): ~2KB
  • Theta (k=4096): ~32KB