kafka_gen_stage v2.0.1 KafkaGenStage.ConsumerLogic View Source

Separates pure logic of consumer genstage to be unit tested.

Link to this section Summary

Types

Acking to prod consumer

Internal type for working with demand and event buffers

Last offset (inclusive) to be emmited by GenStage

Format of read messages

API type for working with demand and event buffers

Functions

Insert incoming messages into buffer, up to end_offset(inclusive)

Handles caching of both, demand and events, as recomended in Buffering Demand in gen_stage docs. Thus called from both, KafkaGenStage.Consumer.handle_demand/2 and KafkaGenStage.Consumer.handle_info/2 where events from brod_consumer arrive

Link to this section Types

Acking to prod consumer.

Link to this type dispatch() View Source
dispatch() ::
  {msgs_to_send :: [msg_tuple()], ack_to_brod_consumer :: ack(),
   buffered_msgs :: :queue.queue()}

Internal type for working with demand and event buffers.

Link to this type end_offset() View Source
end_offset() :: non_neg_integer() | :infinity

Last offset (inclusive) to be emmited by GenStage.

Link to this type msg_tuple() View Source
msg_tuple() ::
  {offset :: non_neg_integer(), timestamp :: non_neg_integer(), key :: binary(),
   value :: binary()}

Format of read messages.

Link to this type return_dispatch() View Source
return_dispatch() ::
  {msgs_to_send :: [msg_tuple()], ack_to_brod_consumer :: ack(),
   buffered_demand :: non_neg_integer(), buffered_msgs :: :queue.queue()}

API type for working with demand and event buffers.

Link to this section Functions

Link to this function messages_into_queue(messages, queue, end_offset, msg_to_tuple \\ &(&1)) View Source
messages_into_queue(
  messages :: [any()],
  buffered_message :: :queue.queue(msg_tuple()),
  inclusive_end_offset :: end_offset(),
  (msg :: term() -> msg_tuple())
) :: {:cont | :halt, :queue.queue(msg_tuple())}

Insert incoming messages into buffer, up to end_offset(inclusive)

Link to this function prepare_dispatch(queue, demand, bulk_transformer, is_end_of_stream) View Source
prepare_dispatch(
  buffered_msgs :: :queue.queue(),
  buffered_demand :: non_neg_integer(),
  bulk_transformer :: ([msg_tuple()] -> [msg_tuple()]),
  is_end_of_stream :: boolean()
) :: return_dispatch()

Handles caching of both, demand and events, as recomended in Buffering Demand in gen_stage docs. Thus called from both, KafkaGenStage.Consumer.handle_demand/2 and KafkaGenStage.Consumer.handle_info/2 where events from brod_consumer arrive.