Postgrex v0.1.0 KafkaGenStage
A module that consumes data from Kafka and injects them into a GenStage Producer
This allows GenStage Consumer-Producers to use a Kafka topic as a regular GenStage data source.
For the Kafka integration it will use the brod package. It will start a number of clients specified in the start_link.
The subscribers will receive messages from the kafka broker, and send it into the genstage using a “call”.
The messages will be stacked up in the internal state of the GenStage, and delived on demand from GenStage consumers.
Summary
Functions
An API method that stacks a new kafka message in the state, so that it can be picked off there on demand
The GenStage producer function
brod_group_subscriber callback
The init function for the GenStage server
The Group Subscriber init callback
This is where messages from kafka ends up before they are sent into the GenStage and stacked
Starts the KafkaGenStage with the options
Functions
An API method that stacks a new kafka message in the state, so that it can be picked off there on demand.
The GenStage producer function.
It simply picks messages out of state, and removes them from the internal list in state.
brod_group_subscriber callback
Called by the subscriber, this is where brod leaves messages.
This is where messages from kafka ends up before they are sent into the GenStage and stacked.
Starts the KafkaGenStage with the options.
The options handled by the module arei (defaults shown).
:hosts - [localhost: 9092] :topics - [“no_topic”] :group - “kgs_cg” :nof_clients - 2
Given as a keyword lists, i.e.
KafkaGenStage.start_link( topics: [“my_topic”], hosts: [my_broker.example.com: 9092] )