NSQ.Consumer (elixir_nsq v1.2.0)

A consumer is a process that creates connections to NSQD to receive messages for a specific topic and channel. It has three primary functions:

  1. Provide a simple interface for a user to setup and configure message handlers.
  2. Balance RDY across all available connections.
  3. Add/remove connections as they are discovered.

Simple Interface

In standard practice, the only function a user should need to know about is NSQ.Consumer.Supervisor.start_link/3. It takes a topic, a channel, and an NSQ.Config struct, which has possible values defined and explained in nsq/config.ex.

{:ok, consumer} = NSQ.Consumer.Supervisor.start_link("my-topic", "my-channel", %NSQ.Config{
  nsqlookupds: ["127.0.0.1:6751", "127.0.0.1:6761"],
  message_handler: fn(body, msg) ->
    # handle them message
    :ok
  end
})

Message handler return values

The return value of the message handler determines how we will respond to NSQ.

:ok

The message was handled and should not be requeued. This sends a FIN command to NSQD.

:req

This message should be requeued. With no delay specified, it will calculate delay exponentially based on the number of attempts. Refer to Message.calculate_delay for the exact formula.

#### {:req, delay}

This message should be requeued. Use the delay specified. A positive integer is expected.

#### {:req, delay, backoff}

This message should be requeued. Use the delay specified. If backoff is truthy, the consumer will temporarily set RDY to 0 in order to stop receiving messages. It will use a standard strategy to resume from backoff mode.

This type of return value is only meant for exceptional cases, such as internal network partitions, where stopping message handling briefly could be beneficial. Only use this return value if you know what you're doing.

A message handler that throws an unhandled exception will automatically requeue and enter backoff mode.

NSQ.Message.touch(msg)

NSQ.Config has a property called msg_timeout, which configures the NSQD server to wait that long before assuming the message failed and requeueing it. If you expect your message handler to take longer than that, you can call NSQ.Message.touch(msg) from the message handler to reset the server-side timer.

NSQ.Consumer.change_max_in_flight(consumer, max_in_flight)

If you'd like to manually change the max in flight of a consumer, use this function. It will cause the consumer's connections to rebalance to the new value. If the new max_in_flight is smaller than the current messages in flight, it must wait for the existing handlers to finish or requeue before it can fully rebalance.

Summary

Types

A tuple with a string ID (used to target the connection in NSQ.Connection.Supervisor) and a PID of the connection.

A map, but we can be more specific by asserting some entries that should be set for a connection's state map.

A tuple with a host and a port.

Functions

Public function to change max_in_flight for a consumer. The new value will be balanced across connections.

Returns a specification to start this module under a supervisor.

If the event manager is not defined in NSQ.Config, it will be generated. So if you want to attach event handlers on the fly, you can use a syntax like NSQ.Consumer.event_manager(consumer) |> :gen_event.add_handler(MyHandler, [])

NSQ.Consumer.Supervisor.start_link returns the supervisor pid so that we can effectively recover from consumer crashes. This function takes the supervisor pid and returns the consumer pid. We use this for public facing functions so that the end user can simply target the supervisor, e.g. NSQ.Consumer.change_max_in_flight(supervisor_pid, 100). Not for external use.

Called from tests to assert correct consumer state. Not for external use.

On init, we create a connection for each NSQD instance discovered, and set up loops for discovery and RDY redistribution.

Starts a Consumer process, called via the supervisor.

Types

Link to this type

connection()

@type connection() :: {String.t(), pid()}

A tuple with a string ID (used to target the connection in NSQ.Connection.Supervisor) and a PID of the connection.

Link to this type

cons_state()

@type cons_state() :: %{
  conn_sup_pid: pid(),
  config: NSQ.Config.t(),
  conn_info_pid: pid()
}

A map, but we can be more specific by asserting some entries that should be set for a connection's state map.

Link to this type

host_with_port()

@type host_with_port() :: {String.t(), integer()}

A tuple with a host and a port.

@type state() :: %{conn_sup_pid: pid(), config: NSQ.Config.t(), conn_info_pid: pid()}

Functions

Link to this function

change_max_in_flight(sup_pid, new_max_in_flight)

@spec change_max_in_flight(pid(), integer()) :: {:ok, :ok}

Public function to change max_in_flight for a consumer. The new value will be balanced across connections.

Link to this function

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

See Supervisor.

Link to this function

conn_info(sup_pid)

Link to this function

event_manager(sup_pid)

If the event manager is not defined in NSQ.Config, it will be generated. So if you want to attach event handlers on the fly, you can use a syntax like NSQ.Consumer.event_manager(consumer) |> :gen_event.add_handler(MyHandler, [])

@spec get(pid()) :: pid()

NSQ.Consumer.Supervisor.start_link returns the supervisor pid so that we can effectively recover from consumer crashes. This function takes the supervisor pid and returns the consumer pid. We use this for public facing functions so that the end user can simply target the supervisor, e.g. NSQ.Consumer.change_max_in_flight(supervisor_pid, 100). Not for external use.

Link to this function

get_state(cons)

@spec get_state(pid()) :: {:ok, cons_state()}

Called from tests to assert correct consumer state. Not for external use.

Link to this function

init(cons_state)

@spec init(map()) :: {:ok, cons_state()}

On init, we create a connection for each NSQD instance discovered, and set up loops for discovery and RDY redistribution.

Link to this function

start_link(arg)

@spec start_link({String.t() | String.t() | NSQ.Config.t()}) :: {:ok, pid()}
@spec start_link({String.t() | String.t() | NSQ.Config.t() | list()}) :: {:ok, pid()}

Starts a Consumer process, called via the supervisor.

Link to this function

starved?(sup_pid)