ExDataSketch ships with production-oriented Livebooks that demonstrate
real-world usage patterns. Each Livebook is self-contained and executable,
requiring only Mix.install([{:ex_data_sketch, "~> 0.9.0"}]) plus any
integration dependencies noted below.
Recommended Order
The Livebooks are designed to build knowledge incrementally. The recommended reading order is:
Streaming Cardinality -- Start here. Covers
from_enumerable/2,ExDataSketch.Stream,Collectable, precision tradeoffs, and ULL vs HLL comparison. No extra dependencies.Persistence Snapshots -- How to save, load, and merge sketches using ETS, DETS, and binary serialization. No extra dependencies.
Distributed Merges -- Associativity, commutativity, tree aggregation, and ETS-sharded patterns. No extra dependencies.
Rolling Telemetry -- Time-windowed sketch accumulation, persistence, and
:telemetryevent attachment. No extra dependencies.Broadway Integration --
accumulate/3,accumulate_into/3,PeriodicAggregator. Requires:broadway.GenStage Aggregation --
SketchConsumer,SketchProducer, flushing, and callbacks. No extra dependencies.LiveDashboard Integration -- Wiring
:telemetryevents into Phoenix LiveDashboard metrics. No extra dependencies (Phoenix patterns are documented in comments).Phoenix Observability -- Endpoint cardinality, latency percentiles, Theta set difference, ETS persistence, memory comparison. No extra dependencies.
AI Token Stream Analytics -- Multi-dimensional dashboard combining HLL, ULL, DDSketch, MisraGries, CMS, and Bloom for LLM workload monitoring. No extra dependencies.
What Each Livebook Teaches
| Livebook | Core API | Key Concept |
|---|---|---|
| Streaming Cardinality | Stream.hll, reduce_into, reduce_partitioned, Collectable | Lazy stream consumption, precision/memory tradeoff |
| Persistence Snapshots | Storage.ETS.save/load/merge, Storage.DETS, serialize/1 | Durability hierarchy, EXSK v2 binary format |
| Distributed Merges | merge_many/1, merge/2, Storage.ETS.merge/3 | Associativity, commutativity, tree aggregation |
| Rolling Telemetry | Telemetry.execute, Telemetry.span, GenServer timer | Time-windowed aggregation, periodic flush |
| Broadway Integration | ExDataSketch.Broadway.accumulate/3, ExDataSketch.Broadway.PeriodicAggregator | Batch aggregation, periodic flush, partition handling |
| GenStage Aggregation | SketchConsumer, SketchProducer, flush/1 | Back-pressure, push-based accumulation, callbacks |
| LiveDashboard Integration | Telemetry.event_name, all_event_names, :telemetry.attach | Phoenix metrics wiring, custom dashboard pages |
| Phoenix Observability | HLL, REQ, CMS, Theta, ETS | Per-endpoint DAU, latency distributions, rate limiting |
| AI Token Analytics | HLL, ULL, DDSketch, KLL, MisraGries, CMS, Bloom | Multi-dimensional sketch dashboard |
Running a Livebook
# From the project root
livebook open livebooks/streaming_cardinality.livemd
# Or start Livebook and navigate to the livebooks/ directory
livebook server
Each Livebook begins with a Mix.install cell that fetches the required
dependencies. The Broadway Livebook additionally installs :broadway.
Livebook Listing
| File | Topic | Lines |
|---|---|---|
streaming_cardinality.livemd | Stream/Collectable API, precision, ULL vs HLL | 126 |
persistence_snapshots.livemd | ETS, DETS, serialization, multi-backend | 152 |
distributed_merges.livemd | Associativity, tree aggregation, ETS sharding | 121 |
rolling_telemetry.livemd | Time windows, GenServer, telemetry events | 119 |
broadway_integration.livemd | Batch accumulation, PeriodicAggregator | 130 |
genstage_aggregation.livemd | SketchConsumer, SketchProducer, flushing | 172 |
livedashboard_integration.livemd | Telemetry wiring, custom pages, events | 190 |
phoenix_observability.livemd | DAU, latency, rate limiting, ETS persistence | 263 |
ai_token_analytics.livemd | LLM workload monitoring, multi-sketch dashboard | 192 |