View Source OffBroadwayRedisStream.Producer (off_broadway_redis_stream v0.6.0)

A GenStage Producer for Redis Stream.

Broadway producer acts as a consumer in the specified Redis stream consumer group. You can run multiple consumers to get better throughput and fault tolerance. Please check Redis Stream Intro for details on stream data type.

It supports failover by automatically claiming pending messages when a node dies. A node is considered dead when it fails to send heartbeats.

Currently, it only supports Redis 6.0.2 and above

Producer Options

  • :redis_client_opts - Required. Redis client specific options. Default client is Redix and for Redix this is used to start redix process Redix.start_link(opts). see Redix Documentation

  • :receive_interval - Optional. The duration (in milliseconds) for which the producer waits before making a request for more messages if there are no events in stream. Default is 1000.

  • :stream - Required. Redis stream name

  • :group - Required. Redis consumer group. Group will be created with :group_start_id ID if it is not already present.

  • :group_start_id - Optional. Starting stream ID which should be used when consumer group created. Use $ for latest ID. see XGROUP CREATE. Default is $

  • :consumer_name - Required. Redis consumer name for the Broadway instance in the consumer-group. If you are running multiple consumers, make sure that each consumer has unique name.

  • :heartbeat_interval - Optional. Producer sends heartbeats at regular intervals, this is interval duration. Default is 5000

  • :allowed_missed_heartbeats - Optional. Number of allowed missing heartbeats for a consumer. The consumer is considered to be dead after this and other consumers claim its pending messages. Default is 3

  • :make_stream - Optional. Appends MKSTREAM subcommand to XGROUP CREATE which automatically create the stream if it doesn't exist. See XGROUP CREATE. Default is false

  • delete_on_acknowledgment - Optional. When XACKing a message also XDELete it. Defaults to false

Acknowledgements

Both successful and failed messages are acknowledged by default. Use Broadway.Message.configure_ack/2 to change this behaviour for failed messages. If a message configured to retry, that message will be attempted again in next batch.

if message.metadata.attempt < @max_attempts do
  Message.configure_ack(message, retry: true)
else
  message
end

attempt field in metadata can be used to control maximum retries. use handle_failure callback to handle failures by moving messages to other stream or persisting failed jobs etc

Message Data

Message data is a 2 element list. First item is id of the message, second is the data

Retry After

If message.metadata.retry_after is present and represents a unix timestamp (in milliseconds) in the future then the message will not be retried until after the time has passed. This is useful for when you are using off_broadway_redis_stream to talk to third party apis that may rate limit you and you need to back off from sending requests.