kafka_gen_stage v2.0.4 KafkaGenStage.ConsumerLogic View Source

Separates pure logic of consumer genstage to be unit tested.

Link to this section Summary

Types

Acking to prod consumer

Last offset (inclusive) to be emmited by GenStage

Format of read messages

Functions

Enqueue incoming messages up to end_offset(inclusive)

Handles caching of both, demand and events, as recomended in Buffering Demand in gen_stage docs. Thus called from both, KafkaGenStage.Consumer.handle_demand/2 and KafkaGenStage.Consumer.handle_info/2 where events from brod_consumer arrive

Link to this section Types

Acking to prod consumer.

Link to this type end_offset() View Source
end_offset() :: non_neg_integer() | :infinity

Last offset (inclusive) to be emmited by GenStage.

Link to this type msg_tuple() View Source
msg_tuple() ::
  {offset :: non_neg_integer(), timestamp :: non_neg_integer(), key :: binary(),
   value :: binary()}

Format of read messages.

Link to this section Functions

Link to this function enqueue(queue, messages, end_offset) View Source
enqueue(
  queue :: :queue.queue(msg_tuple()),
  messages :: Enum.t(),
  inclusive_end_offset :: end_offset()
) :: :queue.queue(msg_tuple())

Enqueue incoming messages up to end_offset(inclusive)

Link to this function prepare_dispatch(queue, demand, bulk_transformer, is_end_of_stream, eos_queue) View Source
prepare_dispatch(
  buffered_msgs :: :queue.queue(),
  buffered_demand :: non_neg_integer(),
  bulk_transformer :: ([msg_tuple()], boolean() -> [msg_tuple()]),
  is_end_of_stream :: boolean(),
  eos_queue :: :queue.queue()
) ::
  {msgs_to_send :: [msg_tuple()], ack_to_brod_consumer :: ack(),
   buffered_demand :: non_neg_integer(), buffered_msgs :: :queue.queue(),
   eos_queue :: :queue.queue()}

Handles caching of both, demand and events, as recomended in Buffering Demand in gen_stage docs. Thus called from both, KafkaGenStage.Consumer.handle_demand/2 and KafkaGenStage.Consumer.handle_info/2 where events from brod_consumer arrive.