View Source OffBroadway.Kafka.Acknowledger (off_broadway_kafka_pipeline v2.0.0)
Implements the Broadway acknowledger behaviour, acking messages back to Kafka once they have been successfully processed.
It stores message ack references in ETS as an ordered set, acknowledging them in the order received rather than the order processed. This ensures that a failure of the Broadway pipeline does not allow messages received later but processed faster to erroneously mark lost messages as acknowledged when they should instead be reprocessed on recovery of the pipeline.
Link to this section Summary
Functions
Acknowledges processed messages to Kafka. Due to Kafka's requirement to maintain message order for proper offset management, concatenates successful and failed messages together and stores the total offset to return for acknowledgement.
Constructs an ack_ref
record for storing the status of message acknowledgement
in ETS.
Adds a set of messages, represented by a contiguous range, to ETS for tracking acknowledgement in the proper order.
Returns a specification to start this module under a supervisor.
Creates an acknowledger GenServer process and links it to the current process.
Link to this section Types
@type ack_ref() :: %{ topic: String.t(), partition: non_neg_integer(), generation_id: non_neg_integer() | nil }
Link to this section Functions
@spec ack(map(), [Broadway.Message.t()], [Broadway.Message.t()]) :: :ok
Acknowledges processed messages to Kafka. Due to Kafka's requirement to maintain message order for proper offset management, concatenates successful and failed messages together and stores the total offset to return for acknowledgement.
@spec ack_ref(Elsa.Message.elsa_message()) :: ack_ref()
Constructs an ack_ref
record for storing the status of message acknowledgement
in ETS.
Adds a set of messages, represented by a contiguous range, to ETS for tracking acknowledgement in the proper order.
Returns a specification to start this module under a supervisor.
See Supervisor
.
Creates an acknowledger GenServer process and links it to the current process.