KafkaExHelpers v0.1.2 KafkaExHelpers View Source

KafkaExHelpers provides helper functions to simplify producing, consuming and interacting with kafka topics

Link to this section Summary

Functions

create_workers spawns a set of KafkaEx workers based on an incoming array of modules

encode_and_publish_message publishes messages using the defined format provided to work with the helper modules. it encodes a Map into a json string and persists it to kafka

creates a json payload from an incoming map

fetches uniq worker ids from a collection of kafka worker module names via kafka_meta/0

collects normalized payloads of uniq consumers, handlers and expected batch sizes

formate a timestamp to a iso formated datetime

Reindexes payload by provided key :request_id by default

looping_fetch_messages/3 is publicly provided but used internally in start_consumers/1 if no messages are found on the topic consuming is delayed 1 second

generate a utc standardized timestamp

genreates an iso8601 now timestamp

Normalizes an outgoing payload

select_partition_by_event_id/2 deterministically selects a partition given the value of a provided string event_id. the string is converted to a char_list and the last value is selected

selects a random value from a set of partions

starts a collection of consumer workers defaulting to KafkaEx.fetch/3 for the message handler

validates a string as a uuid

Link to this section Functions

Link to this function create_workers(workers, handler \\ &KafkaEx.create_worker/2) View Source
create_workers([module], function) :: [pid]

create_workers spawns a set of KafkaEx workers based on an incoming array of modules.

The default action for spawning works is &KafkaEx.create_worker/2, &MockHandler.publish/2 is used for demonstration

Examples

iex> defmodule MockCreate do
  ...>   def kafka_meta do
  ...>     %{worker_id: :mock_handler, partitions: [0], topic: 'Mock.V1'}
  ...>   end
  ...>   def publish(_worker_id, _options) do
  ...>     true
  ...>   end
  ...> end
  ...> KafkaExHelpers.create_workers([MockCreate], &MockCreate.publish/2)
  [true]
Link to this function encode_and_publish_message(payload, reference_id, reference_meta, producer \\ &KafkaEx.produce/4) View Source

encode_and_publish_message publishes messages using the defined format provided to work with the helper modules. it encodes a Map into a json string and persists it to kafka

The default action for publishing a message is &KafkaEx.produce/4, &MockProducer.produce/4 is used for demonstration only

Examples

iex> defmodule MockTopic do
  ...>   def kafka_meta do
  ...>     %{worker_id: :mock_handler, partitions: [0], topic: 'Mock.V1'}
  ...>   end
  ...> end
  ...> defmodule MockProducer do
  ...>   def produce(_topic, _partition, _payload, worker_name: _worker_id) do
  ...>     :ok
  ...>   end
  ...> end
  ...> payload = %{hello: "world"}
  ...> reference_id = payload.hello
  ...> KafkaExHelpers.encode_and_publish_message(payload, reference_id, MockTopic.kafka_meta, &MockProducer.produce/4)
  :ok
Link to this function encode_payload_request(normalized_payload) View Source
encode_payload_request(map) :: String.t

creates a json payload from an incoming map

Link to this function fetch_handler_ids(handlers) View Source
fetch_handler_ids([reference]) :: [atom]

fetches uniq worker ids from a collection of kafka worker module names via kafka_meta/0

Link to this function fetch_worker_ids(topics) View Source
fetch_worker_ids([reference]) :: atom | [atom]

collects normalized payloads of uniq consumers, handlers and expected batch sizes

Link to this function format_time(datetime) View Source
format_time(DateTime.t) :: String.t

formate a timestamp to a iso formated datetime

Examples

iex> timestamp = "2017-03-19T02:48:15.814147Z"
iex> {:ok, time, _} = DateTime.from_iso8601(timestamp)
iex> KafkaExHelpers.format_time(time)
"2017-03-19T02:48:15.814147Z"
Link to this function index_payload_by_request_id(payload, key \\ :request_id) View Source

Reindexes payload by provided key :request_id by default

Examples

iex> request_id = "33c5aa7e-3f35-47bc-883b-33ea0ace89f0"
iex> payload = %{request_id: request_id, other_key: "foo"}
iex> KafkaExHelpers.index_payload_by_request_id(payload)
%{
  "33c5aa7e-3f35-47bc-883b-33ea0ace89f0": [
    %{request_id: "33c5aa7e-3f35-47bc-883b-33ea0ace89f0",
    other_key: "foo"}
  ]
}
Link to this function looping_fetch_messages(partition, w, handler) View Source

looping_fetch_messages/3 is publicly provided but used internally in start_consumers/1 if no messages are found on the topic consuming is delayed 1 second

generate a utc standardized timestamp

genreates an iso8601 now timestamp

Link to this function payload_normalizer(payload, timestamp, uuid) View Source
payload_normalizer(map, String.t, String.t) :: map

Normalizes an outgoing payload

Examples

iex> payload = %{hello: :world}
iex> time = "2017-03-19T02:48:15.814147Z"
iex> uuid = "33c5aa7e-3f35-47bc-883b-33ea0ace89f0"
iex> KafkaExHelpers.payload_normalizer(payload, time, uuid)
%{meta: %{
  requested_at: "2017-03-19T02:48:15.814147Z",
  transaction_id: "33c5aa7e-3f35-47bc-883b-33ea0ace89f0"},
request_body: %{hello: :world}}
Link to this function select_partition_by_event_id(event_id, partitions) View Source
select_partition_by_event_id(String.t, [integer]) :: integer

select_partition_by_event_id/2 deterministically selects a partition given the value of a provided string event_id. the string is converted to a char_list and the last value is selected.

Examples

if there is a single partition it will always be selected

iex> KafkaExHelpers.select_partition_by_event_id("hello_world", [0])
  0

a more complex string will always result in the same partition being selected

iex> partitions = [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18]
  ...> reference_id = "64c0cc33-b37a-4768-8463-4715b4dbadd5"
  ...> KafkaExHelpers.select_partition_by_event_id(reference_id, partitions)
  16
Link to this function select_random_partition(partitions) View Source
select_random_partition([integer]) :: integer

selects a random value from a set of partions

Examples

iex> partitions = [1]
iex> KafkaExHelpers.select_random_partition(partitions)
1
Link to this function start_consumers(topics, handler \\ &KafkaEx.fetch/3) View Source
start_consumers([String.t], (... -> any)) :: {:ok, [pid]}

starts a collection of consumer workers defaulting to KafkaEx.fetch/3 for the message handler

Link to this function validate_uuid(uuid) View Source
validate_uuid(String.t) ::
  {:ok, String.t} |
  {:error, :invalid_uuid}

validates a string as a uuid

Examples

iex> KafkaExHelpers.validate_uuid("33c5aa7e-3f35-47bc-883b-33ea0ace89f0")
{:ok, "33c5aa7e-3f35-47bc-883b-33ea0ace89f0"}

iex> KafkaExHelpers.validate_uuid("some_string")
{:error, :invalid_uuid}