elsa v0.7.1 Elsa.Group.Manager

Defines the GenServer process that coordinates assignment of workers to topics/partitions of a given consumer group. Tracks consumer group state and reinstantiates workers to the last unacknowledged message in the event of failure.

Link to this section Summary

Types

The offset from wthich to begin fetch requests: default = latest

Values to configure the consumer, all are optional

endpoints to connect to kafka brokers

Module that implements the Elsa.Consumer.MessageHandler behaviour

Maximum bytes to fetch in batch of messages: default = 1MB

Max number of milliseconds to wait to wait for broker to collect min_bytes of messages: default = 10_000 ms

Minimum bytes to fetch in batch of messages: default = 0

How to reset begin_offset if OffsetOutOfRange exception is received

The total number of bytes allowed to fetch-ahead: default = 100KB

The windows size (number of messages) allowed to fetch-ahead: default = 10

Allow consumer process to sleep this amount of ms if kafka replied with 'empty' messages: default = 1_000 ms

keyword list of config values to start elsa consumer

Functions

Trigger acknowldgement of processed messages back to the cluster.

Trigger acknowledgement of processed messages back to the cluster.

Trigger the assignment of workers to a given topic and partition

Trigger deallocation of all workers from the consumer group and stop worker processes.

Returns a specification to start this module under a supervisor.

Callback implementation for c::brod_group_member.get_committed_offsets/2.

Invoked to handle continue instructions.

Invoked when the server is started. start_link/3 or start/3 will block until it returns.

Start the group manager process and register a name with the process registry.

Link to this section Types

Link to this type

begin_offset()
begin_offset() :: non_neg_integer()

The offset from wthich to begin fetch requests: default = latest

Link to this type

consumer_config()
consumer_config() :: [
  min_bytes: min_bytes(),
  max_bytes: max_bytes(),
  max_wait_time: max_wait_time(),
  sleep_timeout: sleep_timeout(),
  prefetch_count: prefetch_count(),
  prefetch_bytes: prefetch_bytes(),
  begin_offset: begin_offset(),
  offset_reset_policy: offset_reset_policy()
]

Values to configure the consumer, all are optional

Link to this type

endpoints()
endpoints() :: [{hostname(), portnum()}]

endpoints to connect to kafka brokers

Link to this type

handler()
handler() :: module()

Module that implements the Elsa.Consumer.MessageHandler behaviour

Link to this type

hostname()
hostname() :: atom() | String.t()

Link to this type

max_bytes()
max_bytes() :: non_neg_integer()

Maximum bytes to fetch in batch of messages: default = 1MB

Link to this type

max_wait_time()
max_wait_time() :: non_neg_integer()

Max number of milliseconds to wait to wait for broker to collect min_bytes of messages: default = 10_000 ms

Link to this type

min_bytes()
min_bytes() :: non_neg_integer()

Minimum bytes to fetch in batch of messages: default = 0

Link to this type

offset_reset_policy()
offset_reset_policy() :: :reset_to_earliest | :reset_to_latest

How to reset begin_offset if OffsetOutOfRange exception is received

Link to this type

portnum()
portnum() :: pos_integer()

Link to this type

prefetch_bytes()
prefetch_bytes() :: non_neg_integer()

The total number of bytes allowed to fetch-ahead: default = 100KB

Link to this type

prefetch_count()
prefetch_count() :: non_neg_integer()

The windows size (number of messages) allowed to fetch-ahead: default = 10

Link to this type

sleep_timeout()
sleep_timeout() :: non_neg_integer()

Allow consumer process to sleep this amount of ms if kafka replied with 'empty' messages: default = 1_000 ms

Link to this type

start_config()
start_config() :: [
  name: atom(),
  endpoints: endpoints(),
  group: String.t(),
  topics: [topic()],
  handler: handler(),
  handler_init_args: term(),
  config: consumer_config()
]

keyword list of config values to start elsa consumer

Link to this type

topic()
topic() :: String.t()

Link to this section Functions

Link to this function

ack(name, map)
ack(String.t(), %{
  topic: String.t(),
  partition: integer(),
  generation_id: integer(),
  offset: integer()
}) :: :ok

Trigger acknowldgement of processed messages back to the cluster.

Link to this function

ack(name, topic, partition, generation_id, offset)
ack(String.t(), String.t(), integer(), integer(), integer()) :: :ok

Trigger acknowledgement of processed messages back to the cluster.

Link to this function

assignments_received(pid, group, generation_id, assignments)
assignments_received(pid(), term(), integer(), [tuple()]) :: :ok

Trigger the assignment of workers to a given topic and partition

Link to this function

assignments_revoked(pid)
assignments_revoked(pid()) :: :ok

Trigger deallocation of all workers from the consumer group and stop worker processes.

Link to this function

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

See Supervisor.

Link to this function

get_committed_offsets(pid, topic)

Callback implementation for c::brod_group_member.get_committed_offsets/2.

Link to this function

handle_continue(atom, state)

Invoked to handle continue instructions.

It is useful for performing work after initialization or for splitting the work in a callback in multiple steps, updating the process state along the way.

Return values are the same as c:handle_cast/2.

This callback is optional. If one is not implemented, the server will fail if a continue instruction is used.

This callback is only supported on Erlang/OTP 21+.

Callback implementation for GenServer.handle_continue/2.

Invoked when the server is started. start_link/3 or start/3 will block until it returns.

init_arg is the argument term (second argument) passed to start_link/3.

Returning {:ok, state} will cause start_link/3 to return {:ok, pid} and the process to enter its loop.

Returning {:ok, state, timeout} is similar to {:ok, state} except handle_info(:timeout, state) will be called after timeout milliseconds if no messages are received within the timeout.

Returning {:ok, state, :hibernate} is similar to {:ok, state} except the process is hibernated before entering the loop. See c:handle_call/3 for more information on hibernation.

Returning {:ok, state, {:continue, continue}} is similar to {:ok, state} except that immediately after entering the loop the c:handle_continue/2 callback will be invoked with the value continue as first argument.

Returning :ignore will cause start_link/3 to return :ignore and the process will exit normally without entering the loop or calling c:terminate/2. If used when part of a supervision tree the parent supervisor will not fail to start nor immediately try to restart the GenServer. The remainder of the supervision tree will be started and so the GenServer should not be required by other processes. It can be started later with Supervisor.restart_child/2 as the child specification is saved in the parent supervisor. The main use cases for this are:

  • The GenServer is disabled by configuration but might be enabled later.
  • An error occurred and it will be handled by a different mechanism than the Supervisor. Likely this approach involves calling Supervisor.restart_child/2 after a delay to attempt a restart.

Returning {:stop, reason} will cause start_link/3 to return {:error, reason} and the process to exit with reason reason without entering the loop or calling c:terminate/2.

Callback implementation for GenServer.init/1.

Link to this function

start_link(opts)
start_link(start_config()) :: GenServer.on_start()

Start the group manager process and register a name with the process registry.