View Source OffBroadway.Polyn.Producer (Polyn v0.3.0)
A Broadway Producer for Polyn.
The word Producer
here is confusing because the word is overloaded.
In this module Producer
refers to GenStage data
pipelines where a :producer
is the stage that receives demand for data and sends it to a :consumer
.
This module doesn't "produce" new events that get added to the NATS server for other services to consume.
Rather it consumes existing events from a NATS Stream and passes them to GenStage :consumer
modules
in one application.
usage
Usage
This module wraps OffBroadway.Jetstream.Producer
and will validate that any messages coming through
are valid events and conform to the schema for the event. Use the OffBroadway.Jetstream.Producer
documentation
to learn how to use it. One difference is that you will use OffBroadway.Polyn.Producer
in your :module
configuration instead of the Jetstream one. Invalid messages will send an ACKTERM
to the NATS server so that they aren't sent again. They will be marked as failed
and removed from the pipeline.
Valid messages that come in a batch with an invalid message will send a NACK response before an error
is raised so that the NATS server will know they were received but need to be sent again
Another key difference that Polyn adds is that the :consumer_name
will be taken care of for you
by using the passed type
and configured :source_root
. You can pass in a :source
to :module
to get a more specific :consumer_name
.
example
Example
defmodule MyBroadway do
use Broadway
def start_link(_opts) do
Broadway.start_link(
__MODULE__,
name: MyBroadway,
producer: [
module: {
OffBroadway.Polyn.Producer,
connection_name: :gnat,
type: "user.created.v1"
},
concurrency: 10
],
processors: [
default: [concurrency: 10]
],
batchers: [
example: [
concurrency: 5,
batch_size: 10,
batch_timeout: 2_000
]
]
)
end
def handle_message(_processor_name, message, _context) do
message
|> Message.update_data(&process_data/1)
|> Message.put_batcher(:example)
end
end