View Source OffBroadway.Polyn.Producer (Polyn v0.1.1)

A Broadway Producer for Polyn.

The word Producer here is confusing because the word is overloaded. In this module Producer refers to GenStage data pipelines where a :producer is the stage that receives demand for data and sends it to a :consumer. This module doesn't "produce" new events that get added to the NATS server for other services to consume. Rather it consumes existing events from a NATS Stream and passes them to GenStage :consumer modules in one application.

usage

Usage

This module wraps OffBroadway.Jetstream.Producer and will validate that any messages coming through are valid events and conform to the schema for the event. Use the OffBroadway.Jetstream.Producer documentation to learn how to use it. The only difference being you will use OffBroadway.Polyn.Producer in your :module configuration instead of the Jetsteram one. Invalid messages will send an ACKTERM to the NATS server so that they aren't sent again. They will be marked as failed and removed from the pipeline. Valid messages that come in a batch with an invalid message will send a NACK response before an error is raised so that the NATS server will know they were received but need to be sent again

example

Example

defmodule MyBroadway do
  use Broadway

  def start_link(_opts) do
    Broadway.start_link(
      __MODULE__,
      name: MyBroadway,
      producer: [
        module: {
          OffBroadway.Polyn.Producer,
          connection_name: :gnat,
          stream_name: "TEST_STREAM",
          consumer_name: "TEST_CONSUMER"
        },
        concurrency: 10
      ],
      processors: [
        default: [concurrency: 10]
      ],
      batchers: [
        example: [
          concurrency: 5,
          batch_size: 10,
          batch_timeout: 2_000
        ]
      ]
    )
  end

  def handle_message(_processor_name, message, _context) do
    message
    |> Message.update_data(&process_data/1)
    |> Message.put_batcher(:example)
  end
end