off_broadway_kafka v0.2.2 OffBroadway.Kafka.Acknowledger

Implements the Broadway acknowledger behaviour, handling acking of processed messages back to Kafka once they have been successfully processed. Message ack references are stored in ETS as an ordered set and acknowledgements are performed in the order received rather than the order processed to ensure that a failure of the Broadway pipeline does not allow messages received later but processed faster to erroneously mark lost messages as acknowledged when they should instead be reprocessed on recovery of the pipeline.

Link to this section Summary

Functions

Acknowledges processed messages to Kafka. Due to Kafka's requirement to maintain message order for proper offset management, concatenates successful and failed messages together and stores the total offset to return for acknowledgement.

Constructs an ack_ref record for storing the status of message acknowledgement in ETS.

Adds a set of messages, represented by a contiguous range, to ETS for tracking acknowledgement in the proper order.

Returns a specification to start this module under a supervisor.

Creates an acknowledger GenServer process and links it to the current process.

Link to this section Types

Link to this type

ack_ref()
ack_ref() :: %{
  topic: String.t(),
  partition: non_neg_integer(),
  generation_id: non_neg_integer() | nil
}

Link to this section Functions

Link to this function

ack(ack_ref, successful, failed)
ack(map(), [Broadway.Message.t()], [Broadway.Message.t()]) :: :ok

Acknowledges processed messages to Kafka. Due to Kafka's requirement to maintain message order for proper offset management, concatenates successful and failed messages together and stores the total offset to return for acknowledgement.

Link to this function

ack_ref(map)
ack_ref(Elsa.Message.t()) :: ack_ref()

Constructs an ack_ref record for storing the status of message acknowledgement in ETS.

Link to this function

add_offsets(pid, range)
add_offsets(pid(), Range.t()) :: :ok

Adds a set of messages, represented by a contiguous range, to ETS for tracking acknowledgement in the proper order.

Link to this function

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

See Supervisor.

Link to this function

start_link(opts)

Creates an acknowledger GenServer process and links it to the current process.