View Source OffBroadwayRedisStream.Producer (off_broadway_redis_stream v0.6.0)
A GenStage Producer for Redis Stream.
Broadway producer acts as a consumer in the specified Redis stream consumer group. You can run multiple consumers to get better throughput and fault tolerance. Please check Redis Stream Intro for details on stream data type.
It supports failover by automatically claiming pending messages when a node dies. A node is considered dead when it fails to send heartbeats.
Currently, it only supports Redis 6.0.2 and above
Producer Options
:redis_client_opts
- Required. Redis client specific options. Default client is Redix and for Redix this is used to start redix processRedix.start_link(opts)
. see Redix Documentation:receive_interval
- Optional. The duration (in milliseconds) for which the producer waits before making a request for more messages if there are no events in stream. Default is1000
.:stream
- Required. Redis stream name:group
- Required. Redis consumer group. Group will be created with:group_start_id
ID if it is not already present.:group_start_id
- Optional. Starting stream ID which should be used when consumer group created. Use $ for latest ID. see XGROUP CREATE. Default is$
:consumer_name
- Required. Redis consumer name for the Broadway instance in the consumer-group. If you are running multiple consumers, make sure that each consumer has unique name.:heartbeat_interval
- Optional. Producer sends heartbeats at regular intervals, this is interval duration. Default is5000
:allowed_missed_heartbeats
- Optional. Number of allowed missing heartbeats for a consumer. The consumer is considered to be dead after this and other consumers claim its pending messages. Default is3
:make_stream
- Optional. Appends MKSTREAM subcommand toXGROUP CREATE
which automatically create the stream if it doesn't exist. See XGROUP CREATE. Default isfalse
delete_on_acknowledgment
- Optional. WhenXACK
ing a message alsoXDEL
ete it. Defaults tofalse
Acknowledgements
Both successful and failed messages are acknowledged by default. Use
Broadway.Message.configure_ack/2
to change this behaviour for
failed messages. If a message configured to retry, that message will
be attempted again in next batch.
if message.metadata.attempt < @max_attempts do
Message.configure_ack(message, retry: true)
else
message
end
attempt
field in metadata can be used to control maximum retries.
use handle_failure
callback to handle failures by moving messages
to other stream or persisting failed jobs etc
Message Data
Message data is a 2 element list. First item is id of the message, second is the data
Retry After
If message.metadata.retry_after
is present and represents a unix
timestamp (in milliseconds) in the future then the message will not
be retried until after the time has passed. This is useful for when
you are using off_broadway_redis_stream
to talk to third party
apis that may rate limit you and you need to back off from sending
requests.