View Source OffBroadway.Splunk
A Splunk consumer for Broadway.
Broadway producer acts as a consumer for a given Splunk report or (triggered) alert.
When the Broadway pipeline starts, a OffBroadway.Splunk.Queue
process will be started as part of the pipeline
supervision tree. This process is responsible to query the Splunk Web API for available jobs for the given report or
alert. Available jobs will be enqueued and the OffBroadway.Splunk.Producer
process will consume messages
sequentially (from earliest published
to latest) and passing them through the Broadway system.
Read the full documentation here.
installation
Installation
This package is available in Hex, and can be installed
by adding off_broadway_splunk
to your list of dependencies in mix.exs
:
def deps do
[
{:off_broadway_splunk, "~> 2.0"}
]
end
usage
Usage
The OffBroadway.Splunk.SplunkClient
tries to read the following configuration from config.exs
.
# config.exs
config :off_broadway_splunk, :splunk_client,
base_url: System.get_env("SPLUNK_BASE_URL", "https://splunk.example.com"),
api_token: System.get_env("SPLUNK_API_TOKEN", "your-api-token-here")
Options for the OffBroadway.Splunk.SplunkClient
can be configured either in config.exs
or passed as
options directly to the OffBroadway.Splunk.Producer
module. Options are merged, with the passed options
taking precedence over those configured in config.exs
.
# my_broadway.ex
defmodule MyBroadway do
use Broadway
alias Broadway.Message
def start_link(_opts) do
Broadway.start_link(__MODULE__,
name: __MODULE__,
producer: [
module:
{OffBroadway.Splunk.Producer,
name: "My fine report",
config: [api_token: "override-api-token"]}
],
processors: [
default: []
],
batchers: [
default: [
batch_size: 500,
batch_timeout: 5000
]
]
)
end
...callbacks...
end
processing-messages
Processing messages
In order to process incoming messages, we need to implement some callback functions.
defmodule MyBroadway do
use Broadway
alias Broadway.Message
...start_link...
@impl true
def handle_message(_, %Message{data: data} ,_) do
message
|> Message.update_data(fn -> ...whatever... end)
end
@impl true
def handle_batch(_batcher, messages, _batch_info, _context) do
IO.puts("Received a batch of #{length(messages)} messages!")
messages
end
end
For the sake of the example, we're not really doing anything here. Whenever we're receiving a batch of messages, we just prints out a
message saying "Received a batch of messages!", and for each message we run Message.update_data/2
passing a function that can process
that message ie. by doing some calculations on the data or something else.