KafkaExHelpers v0.1.0 KafkaExHelpers View Source
KafkaExHelpers provides helper functions to simplify producing, consuming and interacting with kafka topics
Link to this section Summary
Functions
create_workers spawns a set of KafkaEx
workers based on an incoming array of modules
creates a json payload from an incoming map
fetches uniq worker ids from a collection of kafka worker module names via kafka_meta/0
collects normalized payloads of uniq consumers, handlers and expected batch sizes
formate a timestamp to a iso formated datetime
Reindexes payload by request_id
Examples
iex> request_id = "33c5aa7e-3f35-47bc-883b-33ea0ace89f0"
iex> payload = %{request_id: request_id, other_key: "foo"}
iex> KafkaExHelpers.index_payload_by_request_id(payload)
%{
"33c5aa7e-3f35-47bc-883b-33ea0ace89f0": [
%{request_id: "33c5aa7e-3f35-47bc-883b-33ea0ace89f0",
other_key: "foo"}
]
}
generate a utc standardized timestamp
genreates an iso8601 now timestamp
Normalizes an outgoing payload Examples
iex> payload = %{hello: :world}
iex> time = "2017-03-19T02:48:15.814147Z"
iex> uuid = "33c5aa7e-3f35-47bc-883b-33ea0ace89f0"
iex> KafkaExHelpers.payload_normalizer(payload, time, uuid)
%{meta: %{
requested_at: "2017-03-19T02:48:15.814147Z",
transaction_id: "33c5aa7e-3f35-47bc-883b-33ea0ace89f0"},
request_body: %{hello: :world}}
selects a random value from a set of partions
starts a collection of consumer workers defaulting to KafkaEx.fetch/3 for the message handler
validates a string as a uuid
Examples
iex> KafkaExHelpers.validate_uuid("33c5aa7e-3f35-47bc-883b-33ea0ace89f0")
{:ok, "33c5aa7e-3f35-47bc-883b-33ea0ace89f0"}
Link to this section Functions
create_workers([module], function) :: [pid]
create_workers spawns a set of KafkaEx
workers based on an incoming array of modules.
The default action for spawning works is &KafkaEx.create_worker/2
, &MockHandler.publish/2
is used for demonstration
Examples
iex> defmodule MockCreate do
...> def kafka_meta do
...> %{worker_id: :mock_handler, partitions: [0], topic: 'Mock.V1'}
...> end
...> def publish(_worker_id, _options) do
...> true
...> end
...> end
...> KafkaExHelpers.create_workers([MockCreate], &MockCreate.publish/2)
[true]
encode_payload_request(map) :: String.t
creates a json payload from an incoming map
fetch_handler_ids([reference]) :: [atom]
fetches uniq worker ids from a collection of kafka worker module names via kafka_meta/0
fetch_worker_ids([reference]) :: atom | [atom]
collects normalized payloads of uniq consumers, handlers and expected batch sizes
formate a timestamp to a iso formated datetime
Examples
iex> timestamp = "2017-03-19T02:48:15.814147Z"
iex> {:ok, time, _} = DateTime.from_iso8601(timestamp)
iex> KafkaExHelpers.format_time(time)
"2017-03-19T02:48:15.814147Z"
Reindexes payload by request_id
Examples
iex> request_id = "33c5aa7e-3f35-47bc-883b-33ea0ace89f0"
iex> payload = %{request_id: request_id, other_key: "foo"}
iex> KafkaExHelpers.index_payload_by_request_id(payload)
%{
"33c5aa7e-3f35-47bc-883b-33ea0ace89f0": [
%{request_id: "33c5aa7e-3f35-47bc-883b-33ea0ace89f0",
other_key: "foo"}
]
}
generate a utc standardized timestamp
genreates an iso8601 now timestamp
Normalizes an outgoing payload Examples
iex> payload = %{hello: :world}
iex> time = "2017-03-19T02:48:15.814147Z"
iex> uuid = "33c5aa7e-3f35-47bc-883b-33ea0ace89f0"
iex> KafkaExHelpers.payload_normalizer(payload, time, uuid)
%{meta: %{
requested_at: "2017-03-19T02:48:15.814147Z",
transaction_id: "33c5aa7e-3f35-47bc-883b-33ea0ace89f0"},
request_body: %{hello: :world}}
select_random_partition([integer]) :: integer
selects a random value from a set of partions
Examples
iex> partitions = [1]
iex> KafkaExHelpers.select_random_partition(partitions)
1
start_consumers([String.t], (... -> any)) :: {:ok, [pid]}
starts a collection of consumer workers defaulting to KafkaEx.fetch/3 for the message handler
validates a string as a uuid
Examples
iex> KafkaExHelpers.validate_uuid("33c5aa7e-3f35-47bc-883b-33ea0ace89f0")
{:ok, "33c5aa7e-3f35-47bc-883b-33ea0ace89f0"}
iex> KafkaExHelpers.validate_uuid("some_string")
{:error, :invalid_uuid}