Telemetry Integration Guide

ExDataSketch emits structured telemetry events at meaningful operation boundaries, enabling production observability without impacting hot-path performance.

Why Telemetry Matters

Individual update/2 calls can run at billions per second -- emitting events for each would cripple throughput. Instead, ExDataSketch emits events at compound operation boundaries:

• from_enumerable/2 -- batch ingestion
• merge_many/1 -- bulk merge
• serialize/1 / deserialize/1 -- serialization (HLL only in this release)
• Storage operations -- save, load, merge, delete
• Stream operations -- partition merge

Configuration

Telemetry is enabled by default. Disable globally:

config :ex_data_sketch, telemetry_enabled: false

Disable specific categories:

config :ex_data_sketch, telemetry: [
  sketch: true,
  persistence: true,
  stream: true,
  pipeline: false
]

Event Reference

Sketch Events

Note: The :ingest event's size_bytes measurement reflects the serialized size of the resulting sketch. Since the sketch is fully constructed before this measurement is taken, it is always available regardless of sketch type.

Persistence Events

Note: The :delete event does not include sketch_type because the sketch struct is no longer available at deletion time. The :load event does not include size_bytes because the binary size is only known after deserialization completes.

Stream Events

Note: The :reduce event is a completion signal emitted from Flow.on_trigger/2. Because the reduce runs inside the Flow runtime, no timing measurement is available.

Pipeline Events

Note: The :periodic_flush duration measures time since the previous flush (or process start), not the time taken to perform the flush itself.

All Event Names

To get a list of all event names programmatically:

ExDataSketch.Telemetry.all_event_names()
# => [[:ex_data_sketch, :sketch, :ingest],
#     [:ex_data_sketch, :sketch, :merge],
#     [:ex_data_sketch, :sketch, :serialize],
#     [:ex_data_sketch, :sketch, :deserialize],
#     [:ex_data_sketch, :persistence, :save],
#     [:ex_data_sketch, :persistence, :load],
#     [:ex_data_sketch, :persistence, :merge],
#     [:ex_data_sketch, :persistence, :delete],
#     [:ex_data_sketch, :stream, :reduce],
#     [:ex_data_sketch, :stream, :partition_merge],
#     [:ex_data_sketch, :pipeline, :accumulate],
#     [:ex_data_sketch, :pipeline, :periodic_flush]]

Attaching Handlers

Use :telemetry.attach/4 to listen for events:

:telemetry.attach("my-handler", [:ex_data_sketch, :sketch, :ingest], fn _name, measurements, metadata, _config ->
  Logger.info("Ingested \#{metadata.sketch_type}: \#{measurements.size_bytes} bytes in \#{measurements.duration} ns")
end, nil)

Measurement Details

All duration measurements use native time units (as returned by System.monotonic_time/0). Convert to milliseconds with:

System.convert_time_unit(duration, :native, :millisecond)

The sketch_type metadata field uses atoms: :hll, :cms, :theta, :ull, :kll, :ddsketch, :req, :frequent_items, :misra_gries, :bloom, :quotient, :cqf, :iblt, :cuckoo, :xor_filter, :filter_chain.

The backend metadata field uses atoms: :ets, :dets, :cubdb, :mnesia, :ecto.

OpenTelemetry Integration

When the :opentelemetry_api dependency is available, bridge telemetry events to OTEL spans:

ExDataSketch.Telemetry.OpenTelemetry.setup()

This attaches handlers that create OpenTelemetry spans for each ExDataSketch telemetry event. Call this in your application's start/2 callback.

To disable:

config :ex_data_sketch, :integrations, opentelemetry: false