Livebooks

ExDataSketch ships with production-oriented Livebooks that demonstrate real-world usage patterns. Each Livebook is self-contained and executable, requiring only Mix.install([{:ex_data_sketch, "~> 0.10"}]) plus any integration dependencies noted below.

Per-Family Tutorials (livebooks/sketches/)

One tutorial livebook per sketch family -- the fastest way to learn a specific family's API, accuracy properties, and operational guidance in isolation, before moving on to the cross-cutting Livebooks below. Each generates its own sample data on first run and caches it under System.tmp_dir!(), so re-running a tutorial (or all 16 in sequence) after the first pass is fast.

Recommended Order

The Livebooks are designed to build knowledge incrementally. The recommended reading order is:

1. Streaming Cardinality -- Start here. Covers from_enumerable/2, ExDataSketch.Stream, Collectable, precision tradeoffs, and ULL vs HLL comparison. No extra dependencies.

2. Persistence Snapshots -- How to save, load, and merge sketches using ETS, DETS, and binary serialization. No extra dependencies.

3. Distributed Merges -- Associativity, commutativity, tree aggregation, and ETS-sharded patterns. No extra dependencies.

4. Broadway Integration -- accumulate/3, accumulate_into/3, PeriodicAggregator. Requires :broadway.

5. GenStage Aggregation -- SketchConsumer, SketchProducer, flushing, and callbacks. No extra dependencies.

6. AI Token Stream Analytics -- Multi-dimensional dashboard combining HLL, ULL, DDSketch, MisraGries, CMS, and Bloom for LLM workload monitoring. No extra dependencies.

7. Sketching One Billion Rows -- Capstone case study. Streams a 1BRC-style (station;temperature) dataset without materializing it, combining exact per-station accumulators with ExDataSketch.KLL (quantiles), ExDataSketch.HLL (distinct stations), and ExDataSketch.FrequentItems (heavy hitters) into one WeatherSketch summary; covers partition-local merge equivalence, bounded concurrency via Task.async_stream/3, and measuring approximation error against the exact accumulator. No extra dependencies beyond :gen_stage and :ecto (compile-time only, pulled in by ExDataSketch's integration modules).

For ExDataSketch.Window (time-windowed sketch accumulation), see guides/windowing.md -- it covers everything the former rolling_telemetry.livemd Livebook demonstrated (basic usage, deterministic testing, persistence) in more depth, in prose plus doctested examples rather than a Livebook that could drift out of sync.

For Phoenix LiveDashboard and telemetry specifically, see phoenix_demo/ at the repository root instead of a Livebook -- a minimal, real, runnable Phoenix app with a mounted ExDataSketch.LiveDashboard.Page, ExDataSketch.Telemetry.Metrics.all/1 wired into its own telemetry module, and a live / page backed by two supervised ExDataSketch.Server instances (one of them windowed, so [:ex_data_sketch, :window, :roll] telemetry -- the former rolling_telemetry.livemd's other topic -- shows up live on the dashboard too). It replaces this project's former livedashboard_integration.livemd, phoenix_observability.livemd, and rolling_telemetry.livemd Livebooks. See phoenix_demo/README.md.

What Each Livebook Teaches

Running a Livebook

# From the project root
livebook open livebooks/streaming_cardinality.livemd

# Or start Livebook and navigate to the livebooks/ directory
livebook server

Each Livebook begins with a Mix.install cell that fetches the required dependencies. The Broadway Livebook additionally installs :broadway; Sketching One Billion Rows additionally installs :gen_stage and :ecto (compile-time dependencies of ExDataSketch's integration modules).

Livebook Listing