kafka_gen_stage v2.0.1 KafkaGenStage.Consumer View Source

Producer GenStage for reading from Kafka topic, using Klarna’s Brod.

Note that is its consumer from Kafka’s perspective, but producer from GenStage’s.

Messages

Events emited are in 4-tuple format

@type msg_tuple :: {offset :: non_neg_integer(), timestamp :: non_neg_integer(),
key :: binary(), value :: binary()}

Options

When starting, several options can modify behaviour of GenStage.

  • :begin_offset - where to start reading the topic, defaut to :earliest, or provide exact offset number (inclusive)

  • :read_end_offset - when to stop reading, also stops gen_stage and sent cancel to subscribers possible values:

    • exact integer of last offset to be read(inclusive)
    • :latest - will check what is offset of last message at time when GenStage is initializing
    • :infinity - does not stop reading by offset, default
  • :stats_handler - every second called function with some statistics, type is: (%{count: non_neg_integer(), cursor: non_neg_integer()}, topic() -> :ok), you can use this function for monitoring of throughput etc…

  • :stats_handler_interval - if you dont want stats_hander to be called every second, provide desired interval (milliseconds)

  • :bulk_transformer - optionally bulk of events can be statelessly transformed before sending downstream. Function signature must be: ([msg_tuple], is_end_of_stream) :: [msg_tuple]

  • :gen_stage_producer_options - refer to options passed to underlaying GenStage, usefull for accumulating demand when partition dispatcher is used ([demand: :accumulate]).

  • :partition - one gen_stage reads from single partition, 0 by default

Starting and stopping brod client

Brod client should be either already started (provided as atom or pid) or provided as initializing function.

Closing of brod client is out of scope of this gen_stage. If you want client to be started exclusively for this gen_stage, do it via initialize function, and manage lifecycle on your own, simple example being:

fn -> :brod.start_link_client([{'localhost', 9092}] = _endpoints) end

Link to this section Summary

Types

Brod’s type for where to start reading in kafka topic

Brod client passing or initialization

Function transforming message bulks before sending them downstream

Last offset (inclusive) to be emmited by GenStage

Format of read messages

All the startup option to configure genstage. See @moduledoc

List of startup options

Starting option(see @moduledoc) which defines when to stop reading

Just documentation purposes of internal state typespec

Runtime stats of consumer. Count of messeges received in last second

Function consuming runtime stats -> to sent to StatsD or so

Kafka topic identifier

Functions

Return some running metadata such as current offset position in topic

Start linked Consumer GenStage of topic (with underlying brod consumer). See option type documentation for possible options

Link to this section Types

Brod’s type for where to start reading in kafka topic.

Link to this type brod_client_init() View Source
brod_client_init() :: atom() | pid() | (() -> {:ok, atom() | pid()})

Brod client passing or initialization.

Link to this type bulk_transformer() View Source
bulk_transformer() :: ([msg_tuple()], boolean() -> [msg_tuple()])

Function transforming message bulks before sending them downstream.

Last offset (inclusive) to be emmited by GenStage.

Format of read messages.

Link to this type option() View Source
option() ::
  {:begin_offset, begin_offset()}
  | {:read_end_offset, read_end_offset()}
  | {:gen_stage_producer_options, [GenStage.producer_option()]}
  | {:partition, integer()}
  | {:stats_handler, stats_handler()}
  | {:bulk_transformer, bulk_transformer()}
  | {:stats_handler_interval, pos_integer()}

All the startup option to configure genstage. See @moduledoc.

Link to this type options() View Source
options() :: [option()]

List of startup options.

Link to this type read_end_offset() View Source
read_end_offset() :: :latest | end_offset()

Starting option(see @moduledoc) which defines when to stop reading.

Link to this type state() View Source
state() :: %KafkaGenStage.Consumer.State{
  brod_client: atom() | pid(),
  brod_client_mref: reference(),
  bulk_transformer: bulk_transformer(),
  consumer: pid(),
  consumer_ref: reference(),
  demand: non_neg_integer(),
  end_offset: end_offset(),
  is_end_of_stream: boolean(),
  partition: non_neg_integer(),
  queue: :queue.queue(),
  reading: :halt | :cont,
  stats: stats(),
  stats_handler: stats_handler(),
  stats_handler_interval: pos_integer(),
  topic: topic()
}

Just documentation purposes of internal state typespec.

Link to this type stats() View Source
stats() :: %{count: non_neg_integer(), cursor: non_neg_integer()}

Runtime stats of consumer. Count of messeges received in last second.

Link to this type stats_handler() View Source
stats_handler() :: (stats(), topic() -> :ok)

Function consuming runtime stats -> to sent to StatsD or so.

Kafka topic identifier.

Link to this section Functions

Link to this function get_insight(reader) View Source
get_insight(server :: term()) ::
  {:ok, %{offset_cursor: non_neg_integer(), topic: topic()}}

Return some running metadata such as current offset position in topic.

Link to this macro kafka_message(args \\ []) View Source (macro)
Link to this macro kafka_message(record, args) View Source (macro)
Link to this macro kafka_message_set(args \\ []) View Source (macro)
Link to this macro kafka_message_set(record, args) View Source (macro)
Link to this function start_link(brod_client_init, topic, options \\ [], gen_server_options \\ []) View Source

Start linked Consumer GenStage of topic (with underlying brod consumer). See option type documentation for possible options.