View Source ExRocketmq.Broker (lib_oss v0.1.0)
RocketMQ Broker Client
The broker is the core component of RocketMQ. It handles responsibilities such as message storage, forwarding, and fulfilling consumer queries.
Each broker cluster has a name server that handles registration and discovery. The broker registers topic and queue metadata with the name server, and producers and consumers locate brokers through the name server to send and receive messages.
Within each broker there can be multiple queues, which store messages. For a given topic, it can be deployed with multiple replicated queues across multiple brokers to ensure high availability.
Additionally, the broker offers message filtering and consumption balancing. Message filtering permits only consuming a subset of messages, and consumption balancing can distribute messages among multiple consumer groups according to consumption speed.
Summary
Functions
Returns a specification to start this module under a supervisor.
When one consumer do consume jobs and failed many times, it will send a consumer_send_msg_back
request to broker,
then broker will try to send the message to another consumer.
In the process of sending transactional messages, after the producer completes the local task check, it will send an end_transaction request to the broker to mark the completion of the transaction for that message.
Get consumer list by group name. This method is used in rebalance process.
Get last offset of queue in remote broker.
In most cases, broker processes are started as subprocesses of producers or consumers. For ease of management, they are generally started using dynamic_supervision, while also registering the addr as a key in the registry, making it possible to locate a specific broker process according to its addr.
Callback implementation for GenServer.handle_continue/2
.
The heartbeat mechanism in RocketMQ is used to maintain the connection status between message producers and consumers and the message middleware. It detects whether the connection is normal by periodically sending heartbeat packets and promptly identifies and handles connection exceptions.
Callback implementation for GenServer.init/1
.
In the scenario of sequential message consumption, the consumer needs to exclusively acquire a certain message queue (mq). By invoking this method, the consumer can perform a remote broker lock operation on the mq.
send message to broker and don't wait for response, the request is totally the same as sync_send_message
,
broker actully will send response to client, but client ignore it.
It's kind of waste of network resource, maybe we can improve it in the future.
pull message from broker.
This request can also update queue's offset via sys_flag
in PullMsg.Request.t()
,
so we can use it to update queue's offset while consuming message.
query offset of queue in remote broker. If the queue has never been consumed, a error code 22 will return
Search offset of queue by timestamp. This method is used when consumer want to consume message from a specific timestamp.
start a broker process, we use get_or_new_broker
in most cases, start_link
is used for special cases if needed.
send message to broker and wait for response.
If you want to send message asynchronously, you can wrap this method in a Task
,
so we don't provide a async_send_message
method.
The reverse operation of lock_batch_mq is typically invoked after rebalancing when the message queue (mq) is reassigned from one consumer to another. Prior to that, it is necessary to call this method to unlock the mq.
Set offset of queue in remote broker.
In most cases, we set remote offset using pull_message
request.
Types
Functions
Returns a specification to start this module under a supervisor.
See Supervisor
.
@spec consumer_send_msg_back(pid(), ExRocketmq.Models.ConsumerSendMsgBack.t()) :: :ok | ExRocketmq.Typespecs.error_t()
When one consumer do consume jobs and failed many times, it will send a consumer_send_msg_back
request to broker,
then broker will try to send the message to another consumer.
@spec end_transaction(pid(), ExRocketmq.Models.EndTransaction.t()) :: :ok | ExRocketmq.Typespecs.error_t()
In the process of sending transactional messages, after the producer completes the local task check, it will send an end_transaction request to the broker to mark the completion of the transaction for that message.
The end_transaction request is a one-way request that may fail. If the request fails, the broker will periodically send a check request to the producer to trigger the client to resend the end_transaction request.
@spec get_consumer_list_by_group(pid(), String.t()) :: {:ok, [String.t()]} | ExRocketmq.Typespecs.error_t()
Get consumer list by group name. This method is used in rebalance process.
@spec get_max_offset(pid(), ExRocketmq.Models.GetMaxOffset.t()) :: {:ok, non_neg_integer()} | ExRocketmq.Typespecs.error_t()
Get last offset of queue in remote broker.
get_or_new_broker(broker_name, addr, registry, dynamic_supervisor, notify_pid \\ nil)
View Source@spec get_or_new_broker( ExRocketmq.Typespecs.broker_name(), String.t(), atom(), pid() | atom(), pid() | nil ) :: pid()
In most cases, broker processes are started as subprocesses of producers or consumers. For ease of management, they are generally started using dynamic_supervision, while also registering the addr as a key in the registry, making it possible to locate a specific broker process according to its addr.
If producer or consumer start broker process using get_or_new_broker
,
it must handle the :notify
message sent by the broker process, like this:
def handle_info({:notify, {pkt, broker_pid}} do
# handle pkt
end
Examples
iex> ExRocketmq.Broker.get_or_new_broker("broker0", "localhost:31001", RegistryName, DynamicSupervisorPid, nil)
#PID<0.123.0>
Callback implementation for GenServer.handle_continue/2
.
@spec heartbeat(pid(), ExRocketmq.Models.Heartbeat.t()) :: :ok | ExRocketmq.Typespecs.error_t()
The heartbeat mechanism in RocketMQ is used to maintain the connection status between message producers and consumers and the message middleware. It detects whether the connection is normal by periodically sending heartbeat packets and promptly identifies and handles connection exceptions.
Specifically, the heartbeat mechanism in RocketMQ has the following key points:
Sending heartbeat packets: Message producers and consumers regularly send heartbeat packets to the message middleware. These packets contain necessary information such as the ID and version number of the producer/consumer. This information helps the middleware identify the client sending the heartbeat.
Receiving heartbeat packets: The message middleware periodically receives heartbeat packets from producers and consumers. By receiving these packets, the middleware can determine if the client is online and check the connection status.
Heartbeat timeout detection: The message middleware sets a heartbeat timeout period to check the reception of heartbeat packets. If no heartbeat packet is received within the specified time, the middleware considers the client's connection to be abnormal and takes appropriate action.
for producer, heartbeat packet carries producer data set, for consumer, heartbeat packet carries consumer data set.
Examples
iex> ExRocketmq.Broker.heartbeat(broker, %ExRocketmq.Models.Heartbeat{
...> client_id: "producer_client_id",
...> producer_data_set: [%ExRocketmq.Models.ProducerData{group: "group"}],
...> })
iex> ExRocketmq.Broker.heartbeat(broker, %ExRocketmq.Models.Heartbeat{
...> client_id: "consumer_client_id",
...> consumer_data_set: [
...> %ExRocketmq.Models.ConsumerData{
...> group: "group"
...> consume_type: "CONSUME_PASSIVELY",
...> message_model: "Clustering",
...> consume_from_where: "CONSUME_FROM_LAST_OFFSET",
...> subscription_data_set: [
...> %ExRocketmq.Models.Subscription{
...> class_filter_mode: false,
...> topic: "topic",
...> sub_string: "*",
...> tags_set: [],
...> code_set: [],
...> sub_version: 0,
...> expression_type: "TAG"
...> }
...> ],
...> unit_mode: false
...> }
...> ],
...> })
Callback implementation for GenServer.init/1
.
@spec lock_batch_mq(pid(), ExRocketmq.Models.Lock.Req.t()) :: {:ok, ExRocketmq.Models.Lock.Resp.t()} | ExRocketmq.Typespecs.error_t()
In the scenario of sequential message consumption, the consumer needs to exclusively acquire a certain message queue (mq). By invoking this method, the consumer can perform a remote broker lock operation on the mq.
send message to broker and don't wait for response, the request is totally the same as sync_send_message
,
broker actully will send response to client, but client ignore it.
It's kind of waste of network resource, maybe we can improve it in the future.
@spec pull_message(pid(), ExRocketmq.Models.PullMsg.Request.t()) :: {:ok, ExRocketmq.Models.PullMsg.Response.t()} | ExRocketmq.Typespecs.error_t()
pull message from broker.
This request can also update queue's offset via sys_flag
in PullMsg.Request.t()
,
so we can use it to update queue's offset while consuming message.
@spec query_consumer_offset(pid(), ExRocketmq.Models.QueryConsumerOffset.t()) :: {:ok, non_neg_integer()} | ExRocketmq.Typespecs.error_t()
query offset of queue in remote broker. If the queue has never been consumed, a error code 22 will return
@spec search_offset_by_timestamp(pid(), ExRocketmq.Models.SearchOffset.t()) :: {:ok, non_neg_integer()} | ExRocketmq.Typespecs.error_t()
Search offset of queue by timestamp. This method is used when consumer want to consume message from a specific timestamp.
@spec send_reply_pkt(pid(), ExRocketmq.Remote.Packet.t()) :: :ok
@spec start_link(namesrvs_opts_schema_t()) :: ExRocketmq.Typespecs.on_start()
start a broker process, we use get_or_new_broker
in most cases, start_link
is used for special cases if needed.
Parameters
:broker_name
(String.t/0
) - Required. The name of the broker:remote_opts
(term/0
) - Required. The remote options of the broker, see ExRocketmq.Remote for details:opts
(keyword/0
) - The options for the broker The default value is[]
.
@spec stop(pid()) :: :ok
@spec sync_send_message(pid(), ExRocketmq.Models.SendMsg.Request.t(), binary()) :: {:ok, ExRocketmq.Models.SendMsg.Response.t()} | ExRocketmq.Typespecs.error_t()
send message to broker and wait for response.
If you want to send message asynchronously, you can wrap this method in a Task
,
so we don't provide a async_send_message
method.
@spec unlock_batch_mq(pid(), ExRocketmq.Models.Lock.Req.t()) :: :ok | ExRocketmq.Typespecs.error_t()
The reverse operation of lock_batch_mq is typically invoked after rebalancing when the message queue (mq) is reassigned from one consumer to another. Prior to that, it is necessary to call this method to unlock the mq.
@spec update_consumer_offset(pid(), ExRocketmq.Models.UpdateConsumerOffset.t()) :: :ok | ExRocketmq.Typespecs.error_t()
Set offset of queue in remote broker.
In most cases, we set remote offset using pull_message
request.