off_broadway_kafka v0.3.1 OffBroadway.Kafka behaviour
Implements a macro for easily including the OffBroadway.Kafka library in your application and declaring the callbacks expected by the framework to configure it's Broadway and Kafka configurations. The OffBroadway.Kafka process creates an Elsa group supervisor as part of its setup to manage consumer group interaction with the cluster.
OffBroadway.Kafka behaviour
To define the custom Broadway behaviour for OffBroadway.Kafka:
- Define a
broadway_config/3
function that receives a keyword list of general configuration values as well as a topic and a partition and constructs the necessary keyword list to configure the Broadway processers and any desired batchers and contexts. Define a
kafka_config/1
function that constructs the necessary configuration to pass to Elsa for defining which Kafka brokers to connect to, which topic and partition Broadway should consume from, a name and consumer group name, as well as consumer configs:- prefetch_count
- prefetch_bytes
- begin_offset
Examples
ClassicHandler
This implementation uses Broadway directly and defines the handler to use the OffBroad.Kafka.Producer handler which communicates with the cluster via Elsa:
defmodule ClassicBroadway do
use Broadway
def start_link(opts) do
kafka_config = [
name: :client1,
brokers: [localhost: 9092],
group: "classic",
topics: ["topic1"],
config: [
begin_offset: :earliest
]
]
Broadway.start_link(__MODULE__,
name: __MODULE__,
producers: [
default: [
module: {OffBroadway.Kafka.Producer, kafka_config},
stages: 1
]
],
processors: [
default: [
stages: 1
]
],
context: %{pid: Keyword.get(opts, :pid)}
)
end
def handle_message(processor, message, context) do
IO.inspect(message, label: "message")
send(context.pid, {:message, message})
message
end
end
ShowtimeHandler
This implementation uses the OffBroadway.Kafka behaviour and starts an Elsa group supervisor directly to have the handler create a per-topic, per-partition basis Broadway pipeline on receiving partition assignments from the group coordinator for nested concurrency of processing events:
defmodule ShowtimeBroadway do
use OffBroadway.Kafka
def kafka_config(_opts) do
[
name: :per_partition,
brokers: [localhost: 9092],
group: "per_partition",
topics: ["topic1"],
config: [
prefetch_count: 5,
prefetch_bytes: 0,
begin_offset: :earliest
]
]
end
def broadway_config(opts, topic, partition) do
[
name: :"broadway_per_partition_#{topic}_#{partition}",
processors: [
default: [
stages: 5
]
],
context: %{
pid: Keyword.get(opts, :pid)
}
]
end
def handle_message(processor, message, context) do
IO.inspect(message, label: "message")
send(context.pid, {:message, message})
message
end
end
Link to this section Summary
Functions
Macro to include the setup functionality for OffBroadway.Kafka and start an Elsa consumer group manager for each Broadway pipeline instantiated.
Link to this section Functions
__using__(opts) (macro)
Macro to include the setup functionality for OffBroadway.Kafka and start an Elsa consumer group manager for each Broadway pipeline instantiated.
Link to this section Callbacks
broadway_config(keyword, arg2, non_neg_integer)
broadway_config(keyword(), String.t(), non_neg_integer()) :: keyword()
broadway_config(keyword(), String.t(), non_neg_integer()) :: keyword()