API Reference kafka_batcher v1.0.0
Modules
flowchart TD
S -->|one for all collectors| CM(ConnectionManager)
S -->|reg_name by topic| APS[AccumulatorsPoolSupervisor]
S[Supervisor] -->|reg_name by topic| C(Collector)
APS --> Accumulator0
APS -->|starts Accumulator for each partition| Accumulator1
APS --> Accumulator2
or only one Accumulator in case when collect_by_partition: false
Accumulator process is used to accumulate messages until Accumulator.State will go into "ready to producing" status. There are many conditions to detect this status, which can be configured through kafka_batcher settings. See details how it works in KafkaBatcher.Accumulator.State module
This module implements logic for force pushing current batch to Kafka, without waiting for other conditions (on size and/or interval).
Encapsulates all logic to detect when batch will be ready to producing
Manage dynamic pool of accumulators for the KafkaBatcher.Collector instance
Determines whether we have to produce the batch immediately
Collector must implement add_events/1 callback to receive events.
ErrorNotifier behaviour is used to report errors to external error-monitoring systems.
KafkaBatcher.Behaviours.Producer adds an abstraction level over producer implementations in various Kafka libs. Defines the callbacks that a Kafka producer should implement
KafkaBatcher.Behaviours.TempStorage behaviour is used to implement events saving logic in case when Kafka is not available.
Implementation of collector for incoming events. The collector accumulates events in accordance with a given strategy using accumulators supervised by AccumulatorsPoolSupervisor. The strategy is specified by the following parameters
Part of the KafkaBatcher.Collector implementation not related to GenServer behavior.
Describes the state of KafkaBatcher.Collector and functions working with it
Provides functions for transforming events in different input formats into MessageObject struct
KafkaBatcher configuration processing. All config parameters are described in details in README.md Examples of configs can be found in the files config/test.exs and test/support/collectors/collector_handlers.ex
Abstraction layer over Kafka library.
State of ConnectionManager process
Default implementation of the KafkaBatcher.Behaviours.ErrorNotifier
behaviour.
Contains Kafka message fields
General part of the Kafka producer implementation
An implementation of the KafkaBatcher.Behaviours.Producer for Kaffe
An implementation of the KafkaBatcher.Behaviours.Producer for KafkaEx
PromEx plugin to collect Prometheus metrics of interactions with Kafka. The following metrics are collected here
The root of KafkaBatcher supervision tree Starts Collector & AccumulatorsPoolSupervisor for each configured collector
Implements wrap-functions that are called to save batches when Kafka is unavailable.
The struct used for KafkaBatcher.Behaviours.TempStorage behavior
Default implementation of KafkaBatcher.Behaviours.TempStorage