View Source OffBroadway.Kafka behaviour (off_broadway_kafka_pipeline v2.0.0)

Defines a macro to easily define a Kafka Broadway pipeline in your application, configuring Broadway and Kafka via callbacks.

It starts a Broadway pipeline for each topic and partition for increased concurrency processing events, receiving partition assignments from the group coordinator and starting an Elsa group supervisor for each.

It uses the following callbacks:

  1. kafka_config/1 receives start_link options and returns the Kafka consumer configuration which is passed to Elsa.Supervisor.start_link/1.

  2. broadway_config/3 receives a keyword list of configuration options, a topic and a partition. It returns the keyword list to configure the Broadway processors, batchers and contexts. Called by OffBroadway.Kafka.ShowtimeHandler.

For example:

defmodule ShowtimeBroadway do
  use OffBroadway.Kafka

  def kafka_config(_opts) do
    [
      connection: :per_partition,
      endpoints: [localhost: 9092],
      group_consumer: [
        group: "per_partition",
        topics: ["topic1"],
        config: [
          prefetch_count: 5,
          prefetch_bytes: 0,
          begin_offset: :earliest
        ]
      ]
    ]
  end

  def broadway_config(opts, topic, partition) do
    [
      name: :"broadway_per_partition_#{topic}_#{partition}",
      processors: [
        default: [
          concurrency: 5
        ]
      ],
      context: %{
        pid: Keyword.get(opts, :pid)
      }
    ]
  end

  def handle_message(processor, message, context) do
    send(context.pid, {:message, message})
    message
  end
end

Link to this section Summary

Functions

Macro which starts pipeline for each Elsa consumer group manager instantiated.

Link to this section Callbacks

Link to this callback

broadway_config(keyword, t, non_neg_integer)

View Source
@callback broadway_config(keyword(), String.t(), non_neg_integer()) :: keyword()
@callback kafka_config(term()) :: keyword()

Link to this section Functions

Link to this macro

__using__(opts)

View Source (macro)

Macro which starts pipeline for each Elsa consumer group manager instantiated.