OffBroadwayRedisStream.Producer (off_broadway_redis_stream v0.1.1) View Source

A GenStage Producer for Redis Stream.

It acts as a consumer in the specified Redis consumer group. Introduction to Redis Stream can be found at: https://redis.io/topics/streams-intro.

Support failover by automatically claiming pending messages of a dead node. A node is considered dead when it fails send heartbeats.

Currently, it only supports Redis 6.0.2 and above

Producer Options

  • :redis_client_opts - Required. Redis client specific options. Default client is Redix and for Redix this is used to start redix process Redix.start_link(opts). see Redix Documentation

  • :receive_interval - Optional. The duration (in milliseconds) for which the producer waits before making a request for more messages if there are no events in stream. Default is 1000.

  • :stream - Required. Redis stream name

  • :group - Required. Redis consumer group. Group will be created with :group_start_id ID if it is not already present.

  • :group_start_id - Optional. Starting stream ID which should be used when consumer group created. Use $ for latest ID. see XGROUP CREATE. Default is $

  • :consumer_name - Required. Redis consumer name for the Broadway instance in the consumer-group. If you are running multiple consumers, make sure that each consumer has unique name.

  • :heartbeat_interval - Optional. Producer sends heartbeats at regular intervals, this is interval duration. Default is 5000

  • :allowed_missed_heartbeats - Optional. Number of allowed missing heartbeats for a consumer. The consumer is considered to be dead after this and other consumers claim its pending messages. Default is 3

  • :make_stream - Optional. Appends MKSTREAM subcommand to XGROUP CREATE which automatically create the stream if it doesn't exist. See XGROUP CREATE. Default is false

Acknowledgments

Both successful and failed messages are acknowledged by default. Use Broadway.Message.configure_ack/2 to change this behaviour for failed messages. If a message configured to retry, that message will be attempted again in next batch.

if message.metadata.attempt < @max_attempts do
  Message.configure_ack(message, retry: true)
else
  message
end

attempt field in metadata can be used to control maximum retries. use handle_failure callback to handle failures by moving messages to other stream or persisting failed jobs etc

Message Data

Message data is a 2 element list. First item is id of the message, second is the data