View Source ExRocketmq.Broker (lib_oss v0.1.0)

RocketMQ Broker Client

The broker is the core component of RocketMQ. It handles responsibilities such as message storage, forwarding, and fulfilling consumer queries.

Each broker cluster has a name server that handles registration and discovery. The broker registers topic and queue metadata with the name server, and producers and consumers locate brokers through the name server to send and receive messages.

Within each broker there can be multiple queues, which store messages. For a given topic, it can be deployed with multiple replicated queues across multiple brokers to ensure high availability.

Additionally, the broker offers message filtering and consumption balancing. Message filtering permits only consuming a subset of messages, and consumption balancing can distribute messages among multiple consumer groups according to consumption speed.

Summary

Functions

Returns a specification to start this module under a supervisor.

When one consumer do consume jobs and failed many times, it will send a consumer_send_msg_back request to broker, then broker will try to send the message to another consumer.

In the process of sending transactional messages, after the producer completes the local task check, it will send an end_transaction request to the broker to mark the completion of the transaction for that message.

Get consumer list by group name. This method is used in rebalance process.

Get last offset of queue in remote broker.

In most cases, broker processes are started as subprocesses of producers or consumers. For ease of management, they are generally started using dynamic_supervision, while also registering the addr as a key in the registry, making it possible to locate a specific broker process according to its addr.

The heartbeat mechanism in RocketMQ is used to maintain the connection status between message producers and consumers and the message middleware. It detects whether the connection is normal by periodically sending heartbeat packets and promptly identifies and handles connection exceptions.

Callback implementation for GenServer.init/1.

In the scenario of sequential message consumption, the consumer needs to exclusively acquire a certain message queue (mq). By invoking this method, the consumer can perform a remote broker lock operation on the mq.

send message to broker and don't wait for response, the request is totally the same as sync_send_message, broker actully will send response to client, but client ignore it. It's kind of waste of network resource, maybe we can improve it in the future.

pull message from broker. This request can also update queue's offset via sys_flag in PullMsg.Request.t(), so we can use it to update queue's offset while consuming message.

query offset of queue in remote broker. If the queue has never been consumed, a error code 22 will return

Search offset of queue by timestamp. This method is used when consumer want to consume message from a specific timestamp.

start a broker process, we use get_or_new_broker in most cases, start_link is used for special cases if needed.

send message to broker and wait for response. If you want to send message asynchronously, you can wrap this method in a Task, so we don't provide a async_send_message method.

The reverse operation of lock_batch_mq is typically invoked after rebalancing when the message queue (mq) is reassigned from one consumer to another. Prior to that, it is necessary to call this method to unlock the mq.

Set offset of queue in remote broker. In most cases, we set remote offset using pull_message request.

Types

Link to this type

namesrvs_opts_schema_t()

View Source
@type namesrvs_opts_schema_t() :: [
  broker_name: binary(),
  remote_opts: term(),
  opts: keyword()
]

Functions

@spec broker_name(pid()) :: String.t()

Returns a specification to start this module under a supervisor.

See Supervisor.

Link to this function

consumer_send_msg_back(broker, req)

View Source
@spec consumer_send_msg_back(pid(), ExRocketmq.Models.ConsumerSendMsgBack.t()) ::
  :ok | ExRocketmq.Typespecs.error_t()

When one consumer do consume jobs and failed many times, it will send a consumer_send_msg_back request to broker, then broker will try to send the message to another consumer.

Link to this function

end_transaction(broker, req)

View Source

In the process of sending transactional messages, after the producer completes the local task check, it will send an end_transaction request to the broker to mark the completion of the transaction for that message.

The end_transaction request is a one-way request that may fail. If the request fails, the broker will periodically send a check request to the producer to trigger the client to resend the end_transaction request.

Link to this function

get_consumer_list_by_group(broker, group)

View Source
@spec get_consumer_list_by_group(pid(), String.t()) ::
  {:ok, [String.t()]} | ExRocketmq.Typespecs.error_t()

Get consumer list by group name. This method is used in rebalance process.

Link to this function

get_max_offset(broker, req)

View Source

Get last offset of queue in remote broker.

Link to this function

get_or_new_broker(broker_name, addr, registry, dynamic_supervisor, notify_pid \\ nil)

View Source
@spec get_or_new_broker(
  ExRocketmq.Typespecs.broker_name(),
  String.t(),
  atom(),
  pid() | atom(),
  pid() | nil
) :: pid()

In most cases, broker processes are started as subprocesses of producers or consumers. For ease of management, they are generally started using dynamic_supervision, while also registering the addr as a key in the registry, making it possible to locate a specific broker process according to its addr.

If producer or consumer start broker process using get_or_new_broker, it must handle the :notify message sent by the broker process, like this:

def handle_info({:notify, {pkt, broker_pid}} do
  # handle pkt
end

Examples

iex> ExRocketmq.Broker.get_or_new_broker("broker0", "localhost:31001", RegistryName, DynamicSupervisorPid, nil)
#PID<0.123.0>
Link to this function

handle_continue(atom, state)

View Source

Callback implementation for GenServer.handle_continue/2.

Link to this function

heartbeat(broker, heartbeat)

View Source

The heartbeat mechanism in RocketMQ is used to maintain the connection status between message producers and consumers and the message middleware. It detects whether the connection is normal by periodically sending heartbeat packets and promptly identifies and handles connection exceptions.

Specifically, the heartbeat mechanism in RocketMQ has the following key points:

  • Sending heartbeat packets: Message producers and consumers regularly send heartbeat packets to the message middleware. These packets contain necessary information such as the ID and version number of the producer/consumer. This information helps the middleware identify the client sending the heartbeat.

  • Receiving heartbeat packets: The message middleware periodically receives heartbeat packets from producers and consumers. By receiving these packets, the middleware can determine if the client is online and check the connection status.

  • Heartbeat timeout detection: The message middleware sets a heartbeat timeout period to check the reception of heartbeat packets. If no heartbeat packet is received within the specified time, the middleware considers the client's connection to be abnormal and takes appropriate action.

for producer, heartbeat packet carries producer data set, for consumer, heartbeat packet carries consumer data set.

Examples

iex> ExRocketmq.Broker.heartbeat(broker, %ExRocketmq.Models.Heartbeat{
...>   client_id: "producer_client_id",
...>   producer_data_set: [%ExRocketmq.Models.ProducerData{group: "group"}],
...> })

iex> ExRocketmq.Broker.heartbeat(broker, %ExRocketmq.Models.Heartbeat{
...>   client_id: "consumer_client_id",
...>   consumer_data_set: [
...>     %ExRocketmq.Models.ConsumerData{
...>           group: "group"
...>           consume_type: "CONSUME_PASSIVELY",
...>           message_model: "Clustering",
...>           consume_from_where: "CONSUME_FROM_LAST_OFFSET",
...>           subscription_data_set: [
...>             %ExRocketmq.Models.Subscription{
...>               class_filter_mode: false,
...>               topic: "topic",
...>               sub_string: "*",
...>               tags_set: [],
...>               code_set: [],
...>               sub_version: 0,
...>               expression_type: "TAG"
...>             }
...>           ],
...>           unit_mode: false
...>     }
...>   ],
...> })

Callback implementation for GenServer.init/1.

Link to this function

lock_batch_mq(broker, req)

View Source

In the scenario of sequential message consumption, the consumer needs to exclusively acquire a certain message queue (mq). By invoking this method, the consumer can perform a remote broker lock operation on the mq.

Link to this function

one_way_send_message(broker, req, body)

View Source
@spec one_way_send_message(pid(), ExRocketmq.Models.SendMsg.Request.t(), binary()) ::
  :ok

send message to broker and don't wait for response, the request is totally the same as sync_send_message, broker actully will send response to client, but client ignore it. It's kind of waste of network resource, maybe we can improve it in the future.

Link to this function

pull_message(broker, req)

View Source
@spec pull_message(pid(), ExRocketmq.Models.PullMsg.Request.t()) ::
  {:ok, ExRocketmq.Models.PullMsg.Response.t()} | ExRocketmq.Typespecs.error_t()

pull message from broker. This request can also update queue's offset via sys_flag in PullMsg.Request.t(), so we can use it to update queue's offset while consuming message.

Link to this function

query_consumer_offset(broker, req)

View Source

query offset of queue in remote broker. If the queue has never been consumed, a error code 22 will return

Link to this function

search_offset_by_timestamp(broker, req)

View Source
@spec search_offset_by_timestamp(pid(), ExRocketmq.Models.SearchOffset.t()) ::
  {:ok, non_neg_integer()} | ExRocketmq.Typespecs.error_t()

Search offset of queue by timestamp. This method is used when consumer want to consume message from a specific timestamp.

Link to this function

send_reply_pkt(broker, pkt)

View Source
@spec send_reply_pkt(pid(), ExRocketmq.Remote.Packet.t()) :: :ok

start a broker process, we use get_or_new_broker in most cases, start_link is used for special cases if needed.

Parameters

  • :broker_name (String.t/0) - Required. The name of the broker

  • :remote_opts (term/0) - Required. The remote options of the broker, see ExRocketmq.Remote for details

  • :opts (keyword/0) - The options for the broker The default value is [].

@spec stop(pid()) :: :ok
Link to this function

sync_send_message(broker, req, body)

View Source
@spec sync_send_message(pid(), ExRocketmq.Models.SendMsg.Request.t(), binary()) ::
  {:ok, ExRocketmq.Models.SendMsg.Response.t()} | ExRocketmq.Typespecs.error_t()

send message to broker and wait for response. If you want to send message asynchronously, you can wrap this method in a Task, so we don't provide a async_send_message method.

Link to this function

unlock_batch_mq(broker, req)

View Source
@spec unlock_batch_mq(pid(), ExRocketmq.Models.Lock.Req.t()) ::
  :ok | ExRocketmq.Typespecs.error_t()

The reverse operation of lock_batch_mq is typically invoked after rebalancing when the message queue (mq) is reassigned from one consumer to another. Prior to that, it is necessary to call this method to unlock the mq.

Link to this function

update_consumer_offset(broker, req)

View Source
@spec update_consumer_offset(pid(), ExRocketmq.Models.UpdateConsumerOffset.t()) ::
  :ok | ExRocketmq.Typespecs.error_t()

Set offset of queue in remote broker. In most cases, we set remote offset using pull_message request.