View Source Offbroadway.EventRelay.Producer (offbroadway_eventrelay v0.1.0)

A GenStage producer that continuously receives messages from an EventRelay Subscription/Topic

For a quick getting started on using Broadway with EventRelay, please see the EventRelay Guide.

Example

defmodule BE.Broadway do
  use Broadway

  alias Broadway.Message

  def start_link(_opts) do
    Broadway.start_link(__MODULE__,
      name: __MODULE__,
      producer: [
        module:
          {Offbroadway.EventRelay.Producer,
           subscription_id: "...",
           host: "localhost",
           port: "50051",
           token: "...",
           certfile: "/path/to/clientcertfile.pem",
           keyfile: "/path/to/clientkeyfile.pem",
           cacertfile: "/path/to/cacertfile.pem"},
      ],
      processors: [
        default: [concurrency: 10]
      ]
    )
  end

  def handle_message(_processor_name, message, _context) do
    # Do something with the message/event
    message
  end
end

The above configuration will set up a producer that continuously pulls events from the configured subscription and sends them downstream.

Options

  • :subscription_id - Required. The name of the subscription, including the project. For example, if you project is named "my-project" and your subscription is named "my-subscription", then your subscription name is "projects/my-project/subscriptions/my-subscription".

  • :host - Required. The host for EventRelay The default value is :noop.

  • :port - Required. The port for the GRPC API for EventRelay The default value is :noop.

  • :token - An ApiKey token that has access to the EventRelay Subscription

  • :cacertfile - Path to a PEM formatted CA certificate file

  • :certfile - Path to a PEM formatted client certificate file

  • :keyfile - Path to a PEM formatted client key file

  • :max_number_of_events (pos_integer/0) - The maximum number of events to be fetched per request. The default value is 50.

  • :pull_interval (integer/0) - The duration (in milliseconds) for which the producer waits before making a request for more messages. The default value is 5000.

  • :pull_timeout - The maximum time (in milliseconds) to wait for a response before the pull client returns an error. The default value is :infinity.

Authentication

For specifics of how to setup authetication in EventRelay use the Authentication Guide in the EventRelay Wiki.

See the Options section for authentication configuration options.

Acknowledgements

This producer uses the PullQueuedEventsRequest in EventRelay which locks an event for a specific subscription when it is pulled from the system. That means any client that pulls events for that subscription in the future will not receive that event.

Dealing with failures it is up to the user of this producer to implement your own failure handling via your own acknowledgement handling. See the example below.

transformer: {__MODULE__, :transform, []} in combination with acknowledger: {__MODULE__, :ack_id, :ack_data} is what allows you to setup up your own acknowledger.

In the future EventRelay will support unlocking an event via the GRPC API which would allow you to re-enqueue an event for processing again on failure.

Example

defmodule BE.Broadway do
  use Broadway

  alias Broadway.Message

  def start_link(_opts) do
    Broadway.start_link(__MODULE__,
      name: __MODULE__,
      producer: [
        module:
          {Offbroadway.EventRelay.Producer,
           subscription_id: "...",
           host: "localhost",
           port: "50051",
           token: "...",
           certfile: "/path/to/clientcertfile.pem",
           keyfile: "/path/to/clientkeyfile.pem",
           cacertfile: "/path/to/cacertfile.pem"},
        transformer: {__MODULE__, :transform, []} 
      ],
      processors: [
        default: [concurrency: 10]
      ]
    )
  end

  def transform(message, _opts) do
    %{
      message
      | acknowledger: {__MODULE__, :ack_id, :ack_data}
    }
  end

  def handle_message(_processor_name, message, _context) do
    # Do something with the message/event
    message
  end

  def ack(:ack_id, _successful, _failed) do
    # Handle the failed messages/events here
  end
end

Summary

Functions

Link to this function

prepare_for_start(module, broadway_opts)

View Source
Link to this function

schedule_next_pull(pull_interval)

View Source