elsa v0.10.0 Elsa.Group.Manager
Defines the GenServer process that coordinates assignment of workers to topics/partitions of a given consumer group. Tracks consumer group state and reinstantiates workers to the last unacknowledged message in the event of failure.
Link to this section Summary
Types
Function called for each new assignment
Function called for when assignments have been revoked
The offset from wthich to begin fetch requests: default = latest
Values to configure the consumer, all are optional
Module that implements the Elsa.Consumer.MessageHandler behaviour
Maximum bytes to fetch in batch of messages: default = 1MB
Max number of milliseconds to wait to wait for broker to collect min_bytes of messages: default = 10_000 ms
Minimum bytes to fetch in batch of messages: default = 0
How to reset begin_offset if OffsetOutOfRange exception is received
The total number of bytes allowed to fetch-ahead: default = 100KB
The windows size (number of messages) allowed to fetch-ahead: default = 10
Allow consumer process to sleep this amount of ms if kafka replied with 'empty' messages: default = 1_000 ms
keyword list of config values to start elsa consumer
Functions
Trigger acknowledgement of processed messages back to the cluster.
Trigger acknowledgement of processed messages back to the cluster.
Trigger the assignment of workers to a given topic and partition
Trigger deallocation of all workers from the consumer group and stop worker processes.
Returns a specification to start this module under a supervisor.
Callback implementation for c::brod_group_member.get_committed_offsets/2
.
Invoked to handle continue
instructions.
Invoked when the server is started. start_link/3
or start/3
will
block until it returns.
Start the group manager process and register a name with the process registry.
Link to this section Types
assignment_received_handler()
assignment_received_handler() ::
(group(), Elsa.topic(), Elsa.partition(), generation_id() ->
:ok | {:error, term()})
assignment_received_handler() :: (group(), Elsa.topic(), Elsa.partition(), generation_id() -> :ok | {:error, term()})
Function called for each new assignment
assignments_revoked_handler()
assignments_revoked_handler() :: (() -> :ok)
assignments_revoked_handler() :: (() -> :ok)
Function called for when assignments have been revoked
begin_offset()
begin_offset() :: non_neg_integer()
begin_offset() :: non_neg_integer()
The offset from wthich to begin fetch requests: default = latest
consumer_config()
consumer_config() :: [
min_bytes: min_bytes(),
max_bytes: max_bytes(),
max_wait_time: max_wait_time(),
sleep_timeout: sleep_timeout(),
prefetch_count: prefetch_count(),
prefetch_bytes: prefetch_bytes(),
begin_offset: begin_offset(),
offset_reset_policy: offset_reset_policy()
]
consumer_config() :: [ min_bytes: min_bytes(), max_bytes: max_bytes(), max_wait_time: max_wait_time(), sleep_timeout: sleep_timeout(), prefetch_count: prefetch_count(), prefetch_bytes: prefetch_bytes(), begin_offset: begin_offset(), offset_reset_policy: offset_reset_policy() ]
Values to configure the consumer, all are optional
generation_id()
generation_id() :: pos_integer()
generation_id() :: pos_integer()
group()
group() :: String.t()
group() :: String.t()
handler()
handler() :: module()
handler() :: module()
Module that implements the Elsa.Consumer.MessageHandler behaviour
max_bytes()
max_bytes() :: non_neg_integer()
max_bytes() :: non_neg_integer()
Maximum bytes to fetch in batch of messages: default = 1MB
max_wait_time()
max_wait_time() :: non_neg_integer()
max_wait_time() :: non_neg_integer()
Max number of milliseconds to wait to wait for broker to collect min_bytes of messages: default = 10_000 ms
min_bytes()
min_bytes() :: non_neg_integer()
min_bytes() :: non_neg_integer()
Minimum bytes to fetch in batch of messages: default = 0
offset_reset_policy()
offset_reset_policy() :: :reset_to_earliest | :reset_to_latest
offset_reset_policy() :: :reset_to_earliest | :reset_to_latest
How to reset begin_offset if OffsetOutOfRange exception is received
prefetch_bytes()
prefetch_bytes() :: non_neg_integer()
prefetch_bytes() :: non_neg_integer()
The total number of bytes allowed to fetch-ahead: default = 100KB
prefetch_count()
prefetch_count() :: non_neg_integer()
prefetch_count() :: non_neg_integer()
The windows size (number of messages) allowed to fetch-ahead: default = 10
sleep_timeout()
sleep_timeout() :: non_neg_integer()
sleep_timeout() :: non_neg_integer()
Allow consumer process to sleep this amount of ms if kafka replied with 'empty' messages: default = 1_000 ms
start_config()
start_config() :: [
connection: Elsa.connection(),
endpoints: Elsa.endpoints(),
group: group(),
topics: [Elsa.topic()],
assignment_received_handler: assignment_received_handler(),
assignments_revoked_handler: assignments_revoked_handler(),
handler: handler(),
handler_init_args: term(),
config: consumer_config()
]
start_config() :: [ connection: Elsa.connection(), endpoints: Elsa.endpoints(), group: group(), topics: [Elsa.topic()], assignment_received_handler: assignment_received_handler(), assignments_revoked_handler: assignments_revoked_handler(), handler: handler(), handler_init_args: term(), config: consumer_config() ]
keyword list of config values to start elsa consumer
Link to this section Functions
ack(connection, map)
ack(String.t(), %{
topic: Elsa.topic(),
partition: Elsa.partition(),
generation_id: generation_id(),
offset: integer()
}) :: :ok
ack(String.t(), %{ topic: Elsa.topic(), partition: Elsa.partition(), generation_id: generation_id(), offset: integer() }) :: :ok
Trigger acknowledgement of processed messages back to the cluster.
ack(connection, topic, partition, generation_id, offset)
ack(String.t(), Elsa.topic(), Elsa.partition(), generation_id(), integer()) ::
:ok
ack(String.t(), Elsa.topic(), Elsa.partition(), generation_id(), integer()) :: :ok
Trigger acknowledgement of processed messages back to the cluster.
assignments_received(pid, group_member_id, generation_id, assignments)
assignments_received(pid(), term(), generation_id(), [tuple()]) :: :ok
assignments_received(pid(), term(), generation_id(), [tuple()]) :: :ok
Trigger the assignment of workers to a given topic and partition
assignments_revoked(pid)
assignments_revoked(pid()) :: :ok
assignments_revoked(pid()) :: :ok
Trigger deallocation of all workers from the consumer group and stop worker processes.
brod_received_assignment(args \\ []) (macro)
brod_received_assignment(record, args) (macro)
child_spec(init_arg)
Returns a specification to start this module under a supervisor.
See Supervisor
.
get_committed_offsets(pid, topic)
Callback implementation for c::brod_group_member.get_committed_offsets/2
.
handle_continue(atom, state)
Invoked to handle continue
instructions.
It is useful for performing work after initialization or for splitting the work in a callback in multiple steps, updating the process state along the way.
Return values are the same as c:handle_cast/2
.
This callback is optional. If one is not implemented, the server will fail if a continue instruction is used.
This callback is only supported on Erlang/OTP 21+.
Callback implementation for GenServer.handle_continue/2
.
init(opts)
Invoked when the server is started. start_link/3
or start/3
will
block until it returns.
init_arg
is the argument term (second argument) passed to start_link/3
.
Returning {:ok, state}
will cause start_link/3
to return
{:ok, pid}
and the process to enter its loop.
Returning {:ok, state, timeout}
is similar to {:ok, state}
except handle_info(:timeout, state)
will be called after timeout
milliseconds if no messages are received within the timeout.
Returning {:ok, state, :hibernate}
is similar to {:ok, state}
except the process is hibernated before entering the loop. See
c:handle_call/3
for more information on hibernation.
Returning {:ok, state, {:continue, continue}}
is similar to
{:ok, state}
except that immediately after entering the loop
the c:handle_continue/2
callback will be invoked with the value
continue
as first argument.
Returning :ignore
will cause start_link/3
to return :ignore
and
the process will exit normally without entering the loop or calling
c:terminate/2
. If used when part of a supervision tree the parent
supervisor will not fail to start nor immediately try to restart the
GenServer
. The remainder of the supervision tree will be started
and so the GenServer
should not be required by other processes.
It can be started later with Supervisor.restart_child/2
as the child
specification is saved in the parent supervisor. The main use cases for
this are:
- The
GenServer
is disabled by configuration but might be enabled later. - An error occurred and it will be handled by a different mechanism than the
Supervisor
. Likely this approach involves callingSupervisor.restart_child/2
after a delay to attempt a restart.
Returning {:stop, reason}
will cause start_link/3
to return
{:error, reason}
and the process to exit with reason reason
without
entering the loop or calling c:terminate/2
.
Callback implementation for GenServer.init/1
.
start_link(opts)
start_link(start_config()) :: GenServer.on_start()
start_link(start_config()) :: GenServer.on_start()
Start the group manager process and register a name with the process registry.