Extreme.PersistentSubscription (extreme v1.1.2)

View Source

An asynchronous subscription strategy.

Other subscription methods require stream positions to be persisted by the client (e.g. in :dets or PostgreSQL). Persistent Subscription is a subscription strategy in which details about backpressure, buffer sizes, and stream positions are all held by the EventStore (server).

In a persistent subscription, all communication is asynchronous. When an event is received and processed, it must be acknowledged as processed by ack/3 in order to be considered by the server as processed. The server stores knowledge of which events have been processed by means of checkpoints, so listeners which use persistent subscriptions do not store stream positions themselves.

The system of ack/3s and nack/5s allows listeners to handle events in unconventional ways:

  • concurrent processing: events may be handled by multiple processors at the same time.
  • out of order processing: events may be handled in any order.
  • retry: events which are nack/5-ed with the :retry action, and events which do not receive acknowledgement via ack/3 are retried.
  • message parking: if an event is not acknowledged and reaches its maximum retry count, the message is parked in a parked messages queue. This prevents head-of-line blocking typical of other subscription patterns.
  • competing consumers: multiple consumers may process events without collaboration or gossip between the consumers themselves.

Persistent subscriptions are started with Extreme.connect_to_persistent_subscription/4 expect a cast of each event in the form of {:on_event, event, correlation_id}

A Persistent Subscription must exist before it can be connected to. Persistent Subscriptions can be created by sending the Extreme.Messages.CreatePersistentSubscription message via Extreme.execute/3, by the HTTP API, or in the EventStore dashboard.

Example

defmodule MyPersistentListener do
  use GenServer

  alias Extreme.PersistentSubscription

  def start_link(opts) do
    GenServer.start_link(__MODULE__, opts)
  end

  def init(opts) do
    {:ok, opts}
  end

  def subscribe(listener_proc), do: GenServer.cast(listener_proc, :subscribe)

  def handle_cast(:subscribe, state) do
    {:ok, subscription_pid} =
      MyExtremeClientModule.connect_to_persistent_subscription(
        self(),
        opts.stream,
        opts.group,
        opts.allowed_in_flight_messages
      )

    {:noreply, Map.put(state, :subscription_pid, subscription_pid)}
  end

  def handle_cast({:on_event, event, correlation_id}, state) do
    # .. do the real processing here ..

    :ok = PersistentSubscription.ack(state.subscription_pid, event, correlation_id)

    {:noreply, state}
  end

  def handle_call(:unsubscribe, _from, state) do
    :ok = MyExtremeClientModule.unsubscribe(state.subscription_pid)

    {:reply, :ok, state}
  end

  def handle_info(_, state), do: {:noreply, state}
end

Summary

Types

An event received from a persistent subscription.

An event ID.

Functions

Acknowledges that an event or set of events have been successfully processed.

Returns a specification to start this module under a supervisor.

Acknowledges that an event or set of events could not be handled.

Types

event()

@type event() :: %Extreme.Messages.ResolvedIndexedEvent{event: term(), link: term()}

An event received from a persistent subscription.

event_id()

@type event_id() :: binary()

An event ID.

Either from the :link or :event, depending on if the event is from a projection stream or a normal stream (respectively).

Functions

ack(subscription, event, correlation_id)

@spec ack(pid(), event() | event_id() | [event() | event_id()], binary()) :: :ok

Acknowledges that an event or set of events have been successfully processed.

ack/3 takes any of the following for event ID:

  • a full event, as given in the {:on_event, event, correlation_id} cast
  • the event_id of an event (either from its :link or :event, depending on if the event comes from a projection or a normal stream, respectively)
  • a list of either sort

correlation_id comes from the :on_event cast.

Example

def handle_cast({:on_event, event, correlation_id}, state) do
  # .. do some processing ..

  # when the processing completes successfully:
  :ok = Extreme.PersistentSubscription.ack(state.subscription_pid, event, correlation_id)

  {:noreply, state}
end

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

See Supervisor.

nack(subscription, event, correlation_id, action, message \\ "")

@spec nack(
  pid(),
  event() | event_id() | [event() | event_id()],
  binary(),
  :unknown | :park | :retry | :skip | :stop,
  String.t()
) :: :ok

Acknowledges that an event or set of events could not be handled.

See ack/3 for information on event and correlation_id.

action can be any of the following

  • :unknown
  • :park
  • :retry
  • :skip
  • :stop

The :park action sets aside the event in the Parked Messages queue, which may be replayed via the HTTP API or by button click in the EventStore Persistent Subscriptions dashboard.

When an event reaches the max retry count configured by the :max_retry_count field in Extreme.Messages.CreatePersistentSubscription, the event is parked.

Example

def handle_cast({:on_event, event, correlation_id}, state) do
  # .. do some processing ..

  # in the case that the processing fails and should be retried:
  :ok = Extreme.PersistentSubscription.nack(state.subscription_pid, event, correlation_id, :retry)

  {:noreply, state}
end