API Reference kafka_batcher v1.0.1

Modules

flowchart TD
    S -->|one for all collectors| CM(ConnectionManager)
    S -->|reg_name by topic| APS[AccumulatorsPoolSupervisor]
    S[Supervisor] -->|reg_name by topic| C(Collector)
    APS --> Accumulator0
    APS -->|starts Accumulator for each partition| Accumulator1
    APS --> Accumulator2

or only one Accumulator in case when collect_by_partition: false

Accumulator process is used to accumulate messages until Accumulator.State will go into "ready to producing" status. There are many conditions to detect this status, which can be configured through kafka_batcher settings. See details how it works in KafkaBatcher.Accumulator.State module

This module implements logic for force pushing current batch to Kafka, without waiting for other conditions (on size and/or interval).

Encapsulates all logic to detect when batch will be ready to producing

Manage dynamic pool of accumulators for the KafkaBatcher.Collector instance

Determines whether we have to produce the batch immediately

Collector must implement add_events/1 callback to receive events.

ErrorNotifier behaviour is used to report errors to external error-monitoring systems.

KafkaBatcher.Behaviours.Producer adds an abstraction level over producer implementations in various Kafka libs. Defines the callbacks that a Kafka producer should implement

KafkaBatcher.Behaviours.TempStorage behaviour is used to implement events saving logic in case when Kafka is not available.

Implementation of collector for incoming events. The collector accumulates events in accordance with a given strategy using accumulators supervised by AccumulatorsPoolSupervisor. The strategy is specified by the following parameters

Part of the KafkaBatcher.Collector implementation not related to GenServer behavior.

Describes the state of KafkaBatcher.Collector and functions working with it

Provides functions for transforming events in different input formats into MessageObject struct

KafkaBatcher configuration processing. All config parameters are described in details in README.md Examples of configs can be found in the files config/test.exs and test/support/collectors/collector_handlers.ex

Abstraction layer over Kafka library.

State of ConnectionManager process

Contains Kafka message fields

General part of the Kafka producer implementation

An implementation of the KafkaBatcher.Behaviours.Producer for Kaffe

An implementation of the KafkaBatcher.Behaviours.Producer for KafkaEx

PromEx plugin to collect Prometheus metrics of interactions with Kafka. The following metrics are collected here

The root of KafkaBatcher supervision tree Starts Collector & AccumulatorsPoolSupervisor for each configured collector

Implements wrap-functions that are called to save batches when Kafka is unavailable.

The struct used for KafkaBatcher.Behaviours.TempStorage behavior

Default implementation of KafkaBatcher.Behaviours.TempStorage