ExDataSketch.Telemetry (ExDataSketch v0.9.0)

Structured telemetry event emission for ExDataSketch.

This module provides a unified interface for emitting telemetry events at meaningful operation boundaries within the ExDataSketch library. Individual update/2 calls do not emit events (they can run at billions per second). Instead, events are emitted at batch/compound operations like from_enumerable/2, merge_many/1, serialize/1, deserialize/1, and all storage and pipeline operations.

Configuration

Telemetry can be disabled entirely or per-category:

# Disable all telemetry (default: true)
config :ex_data_sketch, telemetry_enabled: false

# Disable specific categories (default: all true)
config :ex_data_sketch, telemetry: [
  sketch: true,
  persistence: true,
  stream: true,
  pipeline: true
]

When telemetry is disabled, :telemetry.execute/3 is never called, ensuring zero overhead in production.

Event Names

Sketch Events

Event	Measurements	Metadata
`[:ex_data_sketch, :sketch, :ingest]`	`duration`, `size_bytes` (HLL only)	`sketch_type`
`[:ex_data_sketch, :sketch, :merge]`	`duration`, `merge_count`	`sketch_type`
`[:ex_data_sketch, :sketch, :serialize]`	`duration`, `size_bytes`	`sketch_type`
`[:ex_data_sketch, :sketch, :deserialize]`	`duration`, `size_bytes`	`sketch_type`

Note on :ingest measurements: All sketch types emit duration. Only HLL emits the additional size_bytes measurement via its result callback. Other sketch types emit %{duration} only. This is because from_enumerable/2 consumes a lazy stream and the item count is not available without forcing evaluation.

Persistence Events

Event	Measurements	Metadata
`[:ex_data_sketch, :persistence, :save]`	`duration`, `size_bytes`	`sketch_type`, `backend`, `key`
`[:ex_data_sketch, :persistence, :load]`	`duration`	`sketch_type`, `backend`, `key`
`[:ex_data_sketch, :persistence, :merge]`	`duration`	`sketch_type`, `backend`, `key`
`[:ex_data_sketch, :persistence, :delete]`	`duration`	`backend`, `key`

Note on :delete metadata: No sketch_type is available at deletion time because the sketch struct has already been discarded.

Stream Events

Event	Measurements	Metadata
`[:ex_data_sketch, :stream, :reduce]`	(none)	`sketch_type`
`[:ex_data_sketch, :stream, :partition_merge]`	`duration`, `partition_count`	`sketch_type`

Note on :reduce measurements: The Flow.reduce/3 integration emits this event as a completion signal from Flow.on_trigger/2. Because the reduce runs inside the Flow runtime, the timing is not accessible; the event carries no measurements beyond sketch_type in metadata.

Pipeline Events

Event	Measurements	Metadata
`[:ex_data_sketch, :pipeline, :accumulate]`	`duration`, `count`	`sketch_type`, `batch_size`
`[:ex_data_sketch, :pipeline, :periodic_flush]`	`duration`	`sketch_type`

Note on :periodic_flush duration: The duration measurement represents the time elapsed since the previous flush (or since process start), not the time taken to perform the flush itself.

Usage

Users attach handlers via :telemetry.attach/4:

 :telemetry.attach("my-handler", [:ex_data_sketch, :sketch, :ingest], fn _name, measurements, metadata, _config ->
   Logger.info("Ingested #{metadata.sketch_type}: #{measurements.size_bytes} bytes in #{measurements.duration} ns")
 end, nil)

Or use ExDataSketch.Telemetry.OpenTelemetry.setup/0 to bridge to OpenTelemetry spans when the :opentelemetry_api dependency is available.

Summary

Types

event_name()

measurements()

metadata()

Functions

all_event_names()

Returns all canonical event names.

categories()

Returns all supported event categories.

enabled?(category)

Returns whether telemetry events should be emitted for the given category.

event_name(category, action)

Returns the canonical event name for a given event type.

execute(event_name, measurements, metadata, category)

Emits a telemetry event if telemetry is enabled for the given category.

sketch_type(other)

Returns the sketch type atom for a sketch struct.

sketch_type_from_module(module)

Returns the sketch type atom for a sketch module.

span(event_name, base_measurements, metadata, category, fun)

Executes a function and emits a start/stop telemetry event with duration.

span_with_result(event_name, base_measurements, metadata, category, fun, result_callback)

Emits a telemetry event with timing, returning the result alongside derived measurements.

Types

event_name()

@type event_name() :: [atom(), ...]

measurements()

@type measurements() :: %{required(atom()) => number()}

metadata()

@type metadata() :: %{required(atom()) => term()}

Functions

all_event_names()

@spec all_event_names() :: [event_name()]

Returns all canonical event names.

Examples

iex> length(ExDataSketch.Telemetry.all_event_names()) > 0
true

categories()

@spec categories() :: [atom()]

Returns all supported event categories.

Examples

iex> ExDataSketch.Telemetry.categories()
[:sketch, :persistence, :stream, :pipeline]

enabled?(category)

@spec enabled?(atom()) :: boolean()

Returns whether telemetry events should be emitted for the given category.

Checks the global telemetry_enabled config first, then the per-category config under the :telemetry key.

Examples

iex> is_boolean(ExDataSketch.Telemetry.enabled?(:sketch))
true

event_name(category, action)

@spec event_name(atom(), atom()) :: event_name()

Returns the canonical event name for a given event type.

Useful for attaching handlers programmatically.

Examples

iex> ExDataSketch.Telemetry.event_name(:sketch, :ingest)
[:ex_data_sketch, :sketch, :ingest]

iex> ExDataSketch.Telemetry.event_name(:persistence, :save)
[:ex_data_sketch, :persistence, :save]

execute(event_name, measurements, metadata, category)

@spec execute(event_name(), measurements(), metadata(), atom()) :: :ok

Emits a telemetry event if telemetry is enabled for the given category.

This function checks both the global telemetry_enabled config and the per-category config before emitting. When disabled, it returns :ok immediately without calling :telemetry.execute/3.

Arguments

event_name -- the telemetry event name as a list of atoms.
measurements -- a map of numeric measurements.
metadata -- a map of event metadata.
category -- the event category (:sketch, :persistence, :stream, or :pipeline).

Examples

ExDataSketch.Telemetry.execute(
  [:ex_data_sketch, :sketch, :ingest],
  %{count: 1000, duration: 500_000},
  %{sketch_type: :hll},
  :sketch
)

sketch_type(other)

@spec sketch_type(struct()) :: atom()

Returns the sketch type atom for a sketch struct.

Used in telemetry metadata to identify which sketch type produced an event.

Examples

iex> ExDataSketch.Telemetry.sketch_type(%ExDataSketch.HLL{})
:hll

iex> ExDataSketch.Telemetry.sketch_type(%ExDataSketch.CMS{})
:cms

sketch_type_from_module(module)

@spec sketch_type_from_module(module()) :: atom()

Returns the sketch type atom for a sketch module.

Used internally by storage backends to derive telemetry metadata from a module atom (e.g., ExDataSketch.HLL -> :hll).

Examples

iex> ExDataSketch.Telemetry.sketch_type_from_module(ExDataSketch.HLL)
:hll

iex> ExDataSketch.Telemetry.sketch_type_from_module(ExDataSketch.CMS)
:cms

span(event_name, base_measurements, metadata, category, fun)

@spec span(event_name(), measurements(), metadata(), atom(), (-> result)) :: result
when result: var

Executes a function and emits a start/stop telemetry event with duration.

The event_name is used as-is for the stop event. Measurements include duration in native time units. Base measurements are merged with the computed duration.

Returns the result of fun.

Arguments

event_name -- the telemetry event name.
base_measurements -- a map of pre-computed measurements (can be empty).
metadata -- a map of event metadata.
category -- the event category.
fun -- the zero-arity function to time.

Examples

result = ExDataSketch.Telemetry.span(
  [:ex_data_sketch, :sketch, :merge],
  %{merge_count: 10},
  %{sketch_type: :hll},
  :sketch,
  fn -> ExDataSketch.HLL.merge_many(sketches) end
)

span_with_result(event_name, base_measurements, metadata, category, fun, result_callback)

@spec span_with_result(
  event_name(),
  measurements(),
  metadata(),
  atom(),
  (-> result),
  (result ->
     measurements())
) ::
  result
when result: var

Emits a telemetry event with timing, returning the result alongside derived measurements.

Similar to span/5 but accepts a callback that receives the result and returns additional measurements to merge. This is useful when measurements depend on the result (e.g., size_bytes from HLL.from_enumerable/2).

Arguments

event_name -- the telemetry event name.
base_measurements -- a map of pre-computed measurements.
metadata -- a map of event metadata.
category -- the event category.
fun -- the zero-arity function to time.
result_callback -- a function receiving the result and returning a map of additional measurements to merge.

Examples

{sketch, measurements} = ExDataSketch.Telemetry.span_with_result(
  [:ex_data_sketch, :sketch, :ingest],
  %{},
  %{sketch_type: :hll},
  :sketch,
  fn -> ExDataSketch.HLL.from_enumerable(items, p: 14) end,
  fn sketch -> %{size_bytes: ExDataSketch.HLL.size_bytes(sketch)} end
)