kafka_gen_stage v1.0.1 KafkaGenStage.Consumer View Source
Producer GenStage for reading from Kafka topic, using Klarna’s Brod.
Note that is its consumer from Kafka’s perspective, but producer from GenStage’s.
Messages
Events emited are in 4-tuple format
@type msg_tuple :: {offset :: non_neg_integer(), timestamp :: non_neg_integer(),
key :: binary(), value :: binary()}
Options
When starting, several options can modify behaviour of GenStage.
:begin_offset
- where to start reading the topic, defaut to:earliest
, or provide exact offset number (inclusive):read_end_offset
- when to stop reading, also stops gen_stage and sent cancel to subscribers possible values:- exact integer of last offset to be read(inclusive)
:latest
- will check what is offset of last message at time when GenStage is initializing:infinity
- does not stop reading by offset, default
:stats_handler
- every second called function with some statistics, type is:(%{count: non_neg_integer(), cursor: non_neg_integer()}, topic() -> :ok)
, you can use this function for monitoring of throughput etc…:stats_handler_interval
- if you dont want stats_hander to be called every second, provide desired interval (milliseconds):transformer
- optionally events can be statefully transformed before sending downstream. Function take single msg_tuple and state, returning either single message, or list(possibly empty) of messages and new state. So transformer has flat_map semantics, and if result is list, it is flattened shallowly (not deeply). Errors should be solved internally, so function always return at least {[], state}.:transformer_state
- initial state for final transforming function above.:gen_stage_producer_options
- refer to options passed to underlaying GenStage, usefull for accumulating demand when partition dispatcher is used ([demand: :accumulate]
).:partition
- one gen_stage reads from single partition, 0 by default
Starting and stopping brod client
Brod client should be either already started (provided as atom or pid) or provided as initializing function.
Closing of brod client is out of scope of this gen_stage. If you want client to be started exclusively for this gen_stage, do it via initialize function, and manage lifecycle on your own, simple example being:
fn -> :brod.start_link_client([{'localhost', 9092}] = _endpoints) end
Link to this section Summary
Types
Brod’s type for where to start reading in kafka topic
Brod client passing or initialization
Last offset (inclusive) to be emmited by GenStage
Format of read messages
All the startup option to configure genstage. See @moduledoc
List of startup options
Starting option(see @moduledoc) which defines when to stop reading
Just documentation purposes of internal state typespec
Runtime stats of consumer. Count of messeges received in last second
Function consuming runtime stats -> to sent to StatsD or so
Kafka topic identifier
Msg tuple stateful transforming function called just before emitting event
Functions
Return some running metadata such as current offset position in topic
Start linked Consumer GenStage of topic (with underlying brod consumer). See option type documentation for possible options
Link to this section Types
Brod’s type for where to start reading in kafka topic.
Brod client passing or initialization.
Last offset (inclusive) to be emmited by GenStage.
Format of read messages.
option() :: {:begin_offset, begin_offset()} | {:read_end_offset, read_end_offset()} | {:gen_stage_producer_options, [GenStage.producer_option()]} | {:partition, integer()} | {:stats_handler, stats_handler()} | {:transformer, transformer()} | {:transformer_state, term()} | {:stats_handler_interval, pos_integer()}
All the startup option to configure genstage. See @moduledoc.
List of startup options.
Starting option(see @moduledoc) which defines when to stop reading.
state() :: %KafkaGenStage.Consumer.State{ brod_client: atom() | pid(), brod_client_mref: reference(), consumer: pid(), consumer_ref: reference(), demand: non_neg_integer(), end_offset: end_offset(), partition: non_neg_integer(), queue: :queue.queue(), reading: :halt | :cont, stats: stats(), stats_handler: stats_handler(), stats_handler_interval: pos_integer(), topic: topic(), transformer: transformer(), transformer_state: term() }
Just documentation purposes of internal state typespec.
stats() :: %{count: non_neg_integer(), cursor: non_neg_integer()}
Runtime stats of consumer. Count of messeges received in last second.
Function consuming runtime stats -> to sent to StatsD or so.
Kafka topic identifier.
Msg tuple stateful transforming function called just before emitting event.
Link to this section Functions
get_insight(server :: term()) :: {:ok, %{offset_cursor: non_neg_integer(), topic: topic()}}
Return some running metadata such as current offset position in topic.
start_link(brod_client_init(), topic(), options(), GenServer.options()) :: GenServer.on_start()
Start linked Consumer GenStage of topic (with underlying brod consumer). See option type documentation for possible options.