View Source Offbroadway.EventRelay.Producer (offbroadway_eventrelay v0.2.0)

A GenStage producer that continuously receives messages from an EventRelay Destination/Topic

For a quick getting started on using Broadway with EventRelay, please see the EventRelay Guide.

Example

defmodule BE.Broadway do
  use Broadway

  alias Broadway.Message

  def start_link(_opts) do
    Broadway.start_link(__MODULE__,
      name: __MODULE__,
      producer: [
        module:
          {Offbroadway.EventRelay.Producer,
           destination_id: "...",
           host: "localhost",
           port: "50051",
           token: "...",
           certfile: "/path/to/clientcertfile.pem",
           keyfile: "/path/to/clientkeyfile.pem",
           cacertfile: "/path/to/cacertfile.pem"},
      ],
      processors: [
        default: [concurrency: 10]
      ]
    )
  end

  def handle_message(_processor_name, message, _context) do
    # Do something with the message/event
    message
  end
end

The above configuration will set up a producer that continuously pulls events from the configured destination and sends them downstream.

Options

  • :destination_id - Required. The ID of the destination

  • :host - Required. The host for EventRelay The default value is :noop.

  • :port - Required. The port for the GRPC API for EventRelay The default value is :noop.

  • :token - An ApiKey token that has access to the EventRelay destination

  • :cacertfile - Path to a PEM formatted CA certificate file

  • :certfile - Path to a PEM formatted client certificate file

  • :keyfile - Path to a PEM formatted client key file

  • :pull_interval (integer/0) - The duration (in milliseconds) for which the producer waits before making a request for more messages. The default value is 5000.

  • :pull_timeout - The maximum time (in milliseconds) to wait for a response before the pull client returns an error. The default value is :infinity.

Authentication

For specifics of how to setup authetication in EventRelay use the Authentication Guide in the EventRelay Wiki.

See the Options section for authentication configuration options.

Acknowledgements

This producer uses the PullQueuedEventsRequest in EventRelay which locks an event for a specific destination when it is pulled from the system. That means any client that pulls events for that destination in the future will not receive that event.

Dealing with failures it is up to the user of this producer to implement your own failure handling via your own acknowledgement handling. See the example below.

transformer: {__MODULE__, :transform, []} in combination with acknowledger: {__MODULE__, :ack_id, :ack_data} is what allows you to setup up your own acknowledger.

In the future EventRelay will support unlocking an event via the GRPC API which would allow you to re-enqueue an event for processing again on failure.

Example

defmodule BE.Broadway do
  use Broadway

  alias Broadway.Message

  def start_link(_opts) do
    Broadway.start_link(__MODULE__,
      name: __MODULE__,
      producer: [
        module:
          {Offbroadway.EventRelay.Producer,
           destination_id: "...",
           host: "localhost",
           port: "50051",
           token: "...",
           certfile: "/path/to/clientcertfile.pem",
           keyfile: "/path/to/clientkeyfile.pem",
           cacertfile: "/path/to/cacertfile.pem"},
        transformer: {__MODULE__, :transform, []} 
      ],
      processors: [
        default: [concurrency: 10]
      ]
    )
  end

  def transform(message, _opts) do
    %{
      message
      | acknowledger: {__MODULE__, :ack_id, :ack_data}
    }
  end

  def handle_message(_processor_name, message, _context) do
    # Do something with the message/event
    message
  end

  def ack(:ack_id, _successful, _failed) do
    # Handle the failed messages/events here
  end
end

Summary

Functions

Link to this function

handle_pull_queued_events(error)

View Source
Link to this function

prepare_for_start(module, broadway_opts)

View Source
Link to this function

schedule_next_pull(pull_interval)

View Source