This module implements an API to cover parts of the code with tracing spans that are then exported using the OpenTelemetry protocol.
OpenTelemetry is an observability framework that is widely supported by observability tools.
This module's implementation is based on the opentelemetry-erlang suite of libraries. There is a rudimentary Elixir API there but it's incomplete and non-idiomatic. The idea with this module is to expose all of the functionality we're using in our code by wrapping opentelemetry-erlang's API.
The configuration for OpenTelemetry export is located in config/runtime.exs.
The API implemented here so far includes support for:
Defining a span to cover the execution of a piece of code. See
with_span/3.Propagating span context across Elixir processes, to allow for a span started in one process to be registered as a parent of a span started in a different process. See
get_current_context/1andset_current_context/1.Adding dynamic attributes to the current span, after it has already started. See
add_span_attributes/2.Recording an error or an exception as a span event. See
record_exception/4.
Summary
Types
Span + baggage pair returned by get_current_context/0 and consumed by set_current_context/1.
Functions
Add current-process memory-footprint attributes (see
process_memory_attributes/1) to the current span.
Add dynamic attributes to the current span.
A thin wrapper around :telemetry.execute/3 that adds the span attributes for the current
stack to the metadata.
Removes the current interval timer from prcess memory and returns it.
Retrieve the telemetry span attributes from the persistent term for this stack.
Build a map of current-process memory-footprint attributes suitable for use as span attributes.
Add an error event to the current span.
Set the interval timer for the current process.
Store the telemetry span attributes in the persistent term for this stack.
Records that an interval with the given interval_name has started.
Records the interval timings as attributes in the current span and wipes the interval timer from process memory.
Executes the provided function and records its duration in microseconds.
The duration is added to the current span as a span attribute named with the given name.
Wipe the current interval timer from process memory.
Creates a span providing there is a parent span in the current context.
If there is no parent span, the function fun is called without creating a span.
Create a span that starts at the current point in time and ends when fun returns.
Types
@type otel_ctx() :: {span_ctx() | :undefined, :otel_baggage.t()}
Span + baggage pair returned by get_current_context/0 and consumed by set_current_context/1.
@type span_ctx() :: :opentelemetry.span_ctx()
Functions
@spec add_process_memory_attributes(:start | :end) :: boolean()
Add current-process memory-footprint attributes (see
process_memory_attributes/1) to the current span.
No-op when called outside a span context, avoiding the Process.info/2 cost
on unsampled requests.
Add dynamic attributes to the current span.
For example, if a span is started prior to issuing a DB request, an attribute named
num_rows_fetched can be added to it using this function once the DB query returns its
result.
@spec execute( :telemetry.event_name(), :telemetry.event_measurements() | :telemetry.event_value(), :telemetry.event_metadata() ) :: :ok
A thin wrapper around :telemetry.execute/3 that adds the span attributes for the current
stack to the metadata.
@spec extract_interval_timer() :: Electric.Telemetry.IntervalTimer.t()
Removes the current interval timer from prcess memory and returns it.
Useful if you want to time intervals over multiple processes,
extract the timer, pass it to another process, and then
use set_interval_timer/1 to restore it in the new process.
Retrieve the telemetry span attributes from the persistent term for this stack.
@spec process_memory_attributes(:start | :end) :: %{ required(String.t()) => non_neg_integer() }
Build a map of current-process memory-footprint attributes suitable for use as span attributes.
Captures two values via Process.info/2:
process_bytes— total memory occupied by the process (heap, stack, message queue, GC overhead, etc.)binary_bytes— sum of the sizes of refc binaries referenced by the process; this is what tends to dominate memory in shape requests that buffer large response bodies.
The phase argument determines the attribute key prefix: :start produces
memory.start.process_bytes and memory.start.binary_bytes, :end produces
memory.end.process_bytes and memory.end.binary_bytes.
Caveat: Process.info(self(), :binary) lists one entry per reference, so a
refc binary that the process references multiple times is counted multiple
times in binary_bytes. This is the conventional approximation (Recon's
recon:bin_leak/1 does the same) and is fine for tracking growth across a
span — but it should not be used as a tight per-process memory budget.
:erlang.memory(:binary) gives a deduplicated VM-wide figure if you need
one.
Add an error event to the current span.
@spec set_interval_timer(Electric.Telemetry.IntervalTimer.t()) :: :ok
Set the interval timer for the current process.
@spec set_stack_span_attrs(String.t(), span_attrs()) :: :ok
Store the telemetry span attributes in the persistent term for this stack.
@spec start_interval(atom()) :: :ok
Records that an interval with the given interval_name has started.
This is useful if you want to find out which part of a process took the longest time. It works out simpler than wrapping each part of the process in a timer, and guarentees no gaps in the timings.
Once a number of intervals have been started, call
stop_and_save_intervals() to record the interval timings as
attributes in the current span.
e.g.
OpenTelemetry.start_interval(:quick_sleep.duration_µs)
Process.sleep(1)
OpenTelemetry.start_interval(:longer_sleep.duration_µs)
Process.sleep(2)
OpenTelemetry.stop_and_save_intervals(total_attribute: "total_sleep_µs")will add the following attributes to the current span: quicksleep.durationµs: 1000 longersleep.durationµs: 2000 totalsleepµs: 3000
Records the interval timings as attributes in the current span and wipes the interval timer from process memory.
Options:
:timer- the interval timer to use. If not provided, the timer is extracted from the process memory.:total_attribute- the name of the attribute to store the total time across all intervals. If not provided no total time is recorded.
Executes the provided function and records its duration in microseconds.
The duration is added to the current span as a span attribute named with the given name.
Wipe the current interval timer from process memory.
Creates a span providing there is a parent span in the current context.
If there is no parent span, the function fun is called without creating a span.
This is necessary for the custom way we do sampling, if the parent span is not sampled, the child span will not be created either.
Create a span that starts at the current point in time and ends when fun returns.
Returns the result of calling the function fun.
Calling this function inside another span establishes a parent-child relationship between
the two, as long as both calls happen within the same Elixir process. Use get_current_context/1 for
interprocess progragation of span context.
The stack_id parameter must be set in root spans. For child spans the stack_id is optional
and will be inherited from the parent span.