View Source OffBroadway.Kafka behaviour (off_broadway_kafka_pipeline v2.0.0)
Defines a macro to easily define a Kafka Broadway pipeline in your application, configuring Broadway and Kafka via callbacks.
It starts a Broadway pipeline for each topic and partition for increased concurrency processing events, receiving partition assignments from the group coordinator and starting an Elsa group supervisor for each.
It uses the following callbacks:
kafka_config/1
receivesstart_link
options and returns the Kafka consumer configuration which is passed toElsa.Supervisor.start_link/1
.broadway_config/3
receives a keyword list of configuration options, a topic and a partition. It returns the keyword list to configure the Broadway processors, batchers and contexts. Called byOffBroadway.Kafka.ShowtimeHandler
.
For example:
defmodule ShowtimeBroadway do
use OffBroadway.Kafka
def kafka_config(_opts) do
[
connection: :per_partition,
endpoints: [localhost: 9092],
group_consumer: [
group: "per_partition",
topics: ["topic1"],
config: [
prefetch_count: 5,
prefetch_bytes: 0,
begin_offset: :earliest
]
]
]
end
def broadway_config(opts, topic, partition) do
[
name: :"broadway_per_partition_#{topic}_#{partition}",
processors: [
default: [
concurrency: 5
]
],
context: %{
pid: Keyword.get(opts, :pid)
}
]
end
def handle_message(processor, message, context) do
send(context.pid, {:message, message})
message
end
end
Link to this section Summary
Functions
Macro which starts pipeline for each Elsa consumer group manager instantiated.
Link to this section Callbacks
@callback broadway_config(keyword(), String.t(), non_neg_integer()) :: keyword()
Link to this section Functions
Macro which starts pipeline for each Elsa consumer group manager instantiated.