KafkaExHelpers v0.1.2 KafkaExHelpers View Source
KafkaExHelpers provides helper functions to simplify producing, consuming and interacting with kafka topics
Link to this section Summary
Functions
create_workers spawns a set of KafkaEx
workers based on an incoming array of modules
encode_and_publish_message publishes messages using the defined format provided to work with the helper modules. it encodes a Map into a json string and persists it to kafka
creates a json payload from an incoming map
fetches uniq worker ids from a collection of kafka worker module names via kafka_meta/0
collects normalized payloads of uniq consumers, handlers and expected batch sizes
formate a timestamp to a iso formated datetime
Reindexes payload by provided key :request_id
by default
looping_fetch_messages/3
is publicly provided but used internally in start_consumers/1
if no messages are found on the topic consuming is delayed 1 second
generate a utc standardized timestamp
genreates an iso8601 now timestamp
Normalizes an outgoing payload
select_partition_by_event_id/2
deterministically selects a partition given the value of a provided string event_id. the string is converted to a char_list and the last value is selected
selects a random value from a set of partions
starts a collection of consumer workers defaulting to KafkaEx.fetch/3 for the message handler
validates a string as a uuid
Link to this section Functions
create_workers([module], function) :: [pid]
create_workers spawns a set of KafkaEx
workers based on an incoming array of modules.
The default action for spawning works is &KafkaEx.create_worker/2
, &MockHandler.publish/2
is used for demonstration
Examples
iex> defmodule MockCreate do
...> def kafka_meta do
...> %{worker_id: :mock_handler, partitions: [0], topic: 'Mock.V1'}
...> end
...> def publish(_worker_id, _options) do
...> true
...> end
...> end
...> KafkaExHelpers.create_workers([MockCreate], &MockCreate.publish/2)
[true]
encode_and_publish_message publishes messages using the defined format provided to work with the helper modules. it encodes a Map into a json string and persists it to kafka
The default action for publishing a message is &KafkaEx.produce/4
, &MockProducer.produce/4
is used for demonstration only
Examples
iex> defmodule MockTopic do
...> def kafka_meta do
...> %{worker_id: :mock_handler, partitions: [0], topic: 'Mock.V1'}
...> end
...> end
...> defmodule MockProducer do
...> def produce(_topic, _partition, _payload, worker_name: _worker_id) do
...> :ok
...> end
...> end
...> payload = %{hello: "world"}
...> reference_id = payload.hello
...> KafkaExHelpers.encode_and_publish_message(payload, reference_id, MockTopic.kafka_meta, &MockProducer.produce/4)
:ok
encode_payload_request(map) :: String.t
creates a json payload from an incoming map
fetch_handler_ids([reference]) :: [atom]
fetches uniq worker ids from a collection of kafka worker module names via kafka_meta/0
fetch_worker_ids([reference]) :: atom | [atom]
collects normalized payloads of uniq consumers, handlers and expected batch sizes
formate a timestamp to a iso formated datetime
Examples
iex> timestamp = "2017-03-19T02:48:15.814147Z"
iex> {:ok, time, _} = DateTime.from_iso8601(timestamp)
iex> KafkaExHelpers.format_time(time)
"2017-03-19T02:48:15.814147Z"
Reindexes payload by provided key :request_id
by default
Examples
iex> request_id = "33c5aa7e-3f35-47bc-883b-33ea0ace89f0"
iex> payload = %{request_id: request_id, other_key: "foo"}
iex> KafkaExHelpers.index_payload_by_request_id(payload)
%{
"33c5aa7e-3f35-47bc-883b-33ea0ace89f0": [
%{request_id: "33c5aa7e-3f35-47bc-883b-33ea0ace89f0",
other_key: "foo"}
]
}
looping_fetch_messages/3
is publicly provided but used internally in start_consumers/1
if no messages are found on the topic consuming is delayed 1 second
generate a utc standardized timestamp
genreates an iso8601 now timestamp
Normalizes an outgoing payload
Examples
iex> payload = %{hello: :world}
iex> time = "2017-03-19T02:48:15.814147Z"
iex> uuid = "33c5aa7e-3f35-47bc-883b-33ea0ace89f0"
iex> KafkaExHelpers.payload_normalizer(payload, time, uuid)
%{meta: %{
requested_at: "2017-03-19T02:48:15.814147Z",
transaction_id: "33c5aa7e-3f35-47bc-883b-33ea0ace89f0"},
request_body: %{hello: :world}}
select_partition_by_event_id(String.t, [integer]) :: integer
select_partition_by_event_id/2
deterministically selects a partition given the value of a provided string event_id. the string is converted to a char_list and the last value is selected.
Examples
if there is a single partition it will always be selected
iex> KafkaExHelpers.select_partition_by_event_id("hello_world", [0])
0
a more complex string will always result in the same partition being selected
iex> partitions = [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18]
...> reference_id = "64c0cc33-b37a-4768-8463-4715b4dbadd5"
...> KafkaExHelpers.select_partition_by_event_id(reference_id, partitions)
16
select_random_partition([integer]) :: integer
selects a random value from a set of partions
Examples
iex> partitions = [1]
iex> KafkaExHelpers.select_random_partition(partitions)
1
start_consumers([String.t], (... -> any)) :: {:ok, [pid]}
starts a collection of consumer workers defaulting to KafkaEx.fetch/3 for the message handler
validates a string as a uuid
Examples
iex> KafkaExHelpers.validate_uuid("33c5aa7e-3f35-47bc-883b-33ea0ace89f0")
{:ok, "33c5aa7e-3f35-47bc-883b-33ea0ace89f0"}
iex> KafkaExHelpers.validate_uuid("some_string")
{:error, :invalid_uuid}