KafkaBatcher.PromEx.Plugins.Kafka (kafka_batcher v1.0.1)

PromEx plugin to collect Prometheus metrics of interactions with Kafka. The following metrics are collected here:

  • prom_ex_kafka_producer_batch_total_size_byte_bucket
  • prom_ex_kafka_producer_batch_total_size_byte_sum
  • prom_ex_kafka_producer_batch_total_size_byte_count
  • prom_ex_kafka_producer_batch_messages_count_bucket
  • prom_ex_kafka_producer_batch_messages_count_sum
  • prom_ex_kafka_producer_batch_messages_count_count
  • prom_ex_kafka_producer_duration_seconds_bucket
  • prom_ex_kafka_producer_duration_seconds_sum
  • prom_ex_kafka_producer_duration_seconds_count

Each metric has the following labels:

  • topic (topic name)
  • partition (partition number)
  • topic_alias (short name of topic to improve readability of Grafana dashboards in case when topic

Configuration options that allow you to set metrics display preferences: :kafka_topic_aliases - allows you to set an alias for display in metrics :producer_buckets - allows to set bucket parameters for grouping metrics For example:

 config :kafka_batcher,
   kafka_topic_aliases: %{
     my_topic1 => "topic1",
     my_topic2 => "topic2"
   }

 config :kafka_batcher,
   :kafka_metric_opts,
     producer_buckets:
     [
       duration: [1, 2, 3, 4, 5, 10, 15, 20, 50, 100],
       byte_size: [1_000, 2_000, 5_000, 10_000, 20_000, 50_000, 100_000],
       messages_count: [1, 5, 10, 15, 20, 30, 40, 50, 100]
     ]

Summary

Functions

Link to this function

consumer_event_metrics(metric_prefix, labels, buckets)

Link to this function

producer_event_metrics(metric_prefix, labels, buckets)