Postgrex v0.1.0 KafkaGenStage

A module that consumes data from Kafka and injects them into a GenStage Producer

This allows GenStage Consumer-Producers to use a Kafka topic as a regular GenStage data source.

For the Kafka integration it will use the brod package. It will start a number of clients specified in the start_link.

The subscribers will receive messages from the kafka broker, and send it into the genstage using a “call”.

The messages will be stacked up in the internal state of the GenStage, and delived on demand from GenStage consumers.

Summary

Functions

An API method that stacks a new kafka message in the state, so that it can be picked off there on demand

The GenStage producer function

brod_group_subscriber callback

The init function for the GenStage server

The Group Subscriber init callback

This is where messages from kafka ends up before they are sent into the GenStage and stacked

Starts the KafkaGenStage with the options

Functions

handle_call(msg, from, state)

An API method that stacks a new kafka message in the state, so that it can be picked off there on demand.

handle_demand(nof_wanted, state)

The GenStage producer function.

It simply picks messages out of state, and removes them from the internal list in state.

handle_message(topic, partition, message, state)

brod_group_subscriber callback

Called by the subscriber, this is where brod leaves messages.

init(kafkaOptions)

The init function for the GenStage server.

init(group_id, arg)

The Group Subscriber init callback

kafka_message(args \\ []) (macro)
kafka_message(record, args) (macro)
message_handler_loop(topic, partition, subscriber_pid, genstage_pid)

This is where messages from kafka ends up before they are sent into the GenStage and stacked.

start_link(kafkaOptions)

Starts the KafkaGenStage with the options.

The options handled by the module arei (defaults shown).

:hosts - [localhost: 9092] :topics - [“no_topic”] :group - “kgs_cg” :nof_clients - 2

Given as a keyword lists, i.e.

KafkaGenStage.start_link( topics: [“my_topic”], hosts: [my_broker.example.com: 9092] )