KafkaExHelpers v0.1.0 KafkaExHelpers View Source

KafkaExHelpers provides helper functions to simplify producing, consuming and interacting with kafka topics

Link to this section Summary

Functions

create_workers spawns a set of KafkaEx workers based on an incoming array of modules

creates a json payload from an incoming map

fetches uniq worker ids from a collection of kafka worker module names via kafka_meta/0

collects normalized payloads of uniq consumers, handlers and expected batch sizes

formate a timestamp to a iso formated datetime

Reindexes payload by request_id Examples

iex> request_id = "33c5aa7e-3f35-47bc-883b-33ea0ace89f0"
iex> payload = %{request_id: request_id, other_key: "foo"}
iex> KafkaExHelpers.index_payload_by_request_id(payload)
%{
  "33c5aa7e-3f35-47bc-883b-33ea0ace89f0": [
    %{request_id: "33c5aa7e-3f35-47bc-883b-33ea0ace89f0",
    other_key: "foo"}
  ]
}

generate a utc standardized timestamp

genreates an iso8601 now timestamp

Normalizes an outgoing payload Examples

iex> payload = %{hello: :world}
iex> time = "2017-03-19T02:48:15.814147Z"
iex> uuid = "33c5aa7e-3f35-47bc-883b-33ea0ace89f0"
iex> KafkaExHelpers.payload_normalizer(payload, time, uuid)
%{meta: %{
  requested_at: "2017-03-19T02:48:15.814147Z",
  transaction_id: "33c5aa7e-3f35-47bc-883b-33ea0ace89f0"},
request_body: %{hello: :world}}

selects a random value from a set of partions

starts a collection of consumer workers defaulting to KafkaEx.fetch/3 for the message handler

validates a string as a uuid

Examples

iex> KafkaExHelpers.validate_uuid("33c5aa7e-3f35-47bc-883b-33ea0ace89f0")
{:ok, "33c5aa7e-3f35-47bc-883b-33ea0ace89f0"}

Link to this section Functions

Link to this function create_workers(workers, handler \\ &KafkaEx.create_worker/2) View Source
create_workers([module], function) :: [pid]

create_workers spawns a set of KafkaEx workers based on an incoming array of modules.

The default action for spawning works is &KafkaEx.create_worker/2, &MockHandler.publish/2 is used for demonstration

Examples

iex> defmodule MockCreate do
  ...>   def kafka_meta do
  ...>     %{worker_id: :mock_handler, partitions: [0], topic: 'Mock.V1'}
  ...>   end
  ...>   def publish(_worker_id, _options) do
  ...>     true
  ...>   end
  ...> end
  ...> KafkaExHelpers.create_workers([MockCreate], &MockCreate.publish/2)
  [true]
Link to this function encode_payload_request(normalized_payload) View Source
encode_payload_request(map) :: String.t

creates a json payload from an incoming map

Link to this function fetch_handler_ids(handlers) View Source
fetch_handler_ids([reference]) :: [atom]

fetches uniq worker ids from a collection of kafka worker module names via kafka_meta/0

Link to this function fetch_worker_ids(topics) View Source
fetch_worker_ids([reference]) :: atom | [atom]

collects normalized payloads of uniq consumers, handlers and expected batch sizes

Link to this function format_time(datetime) View Source
format_time(DateTime.t) :: String.t

formate a timestamp to a iso formated datetime

Examples

iex> timestamp = "2017-03-19T02:48:15.814147Z"
iex> {:ok, time, _} = DateTime.from_iso8601(timestamp)
iex> KafkaExHelpers.format_time(time)
"2017-03-19T02:48:15.814147Z"
Link to this function index_payload_by_request_id(payload) View Source

Reindexes payload by request_id Examples

iex> request_id = "33c5aa7e-3f35-47bc-883b-33ea0ace89f0"
iex> payload = %{request_id: request_id, other_key: "foo"}
iex> KafkaExHelpers.index_payload_by_request_id(payload)
%{
  "33c5aa7e-3f35-47bc-883b-33ea0ace89f0": [
    %{request_id: "33c5aa7e-3f35-47bc-883b-33ea0ace89f0",
    other_key: "foo"}
  ]
}
Link to this function looping_fetch_messages(partition, w, handler) View Source

generate a utc standardized timestamp

genreates an iso8601 now timestamp

Link to this function payload_normalizer(payload, timestamp, uuid) View Source
payload_normalizer(map, String.t, String.t) :: map

Normalizes an outgoing payload Examples

iex> payload = %{hello: :world}
iex> time = "2017-03-19T02:48:15.814147Z"
iex> uuid = "33c5aa7e-3f35-47bc-883b-33ea0ace89f0"
iex> KafkaExHelpers.payload_normalizer(payload, time, uuid)
%{meta: %{
  requested_at: "2017-03-19T02:48:15.814147Z",
  transaction_id: "33c5aa7e-3f35-47bc-883b-33ea0ace89f0"},
request_body: %{hello: :world}}
Link to this function select_random_partition(partitions) View Source
select_random_partition([integer]) :: integer

selects a random value from a set of partions

Examples

iex> partitions = [1]
iex> KafkaExHelpers.select_random_partition(partitions)
1
Link to this function start_consumers(topics, handler \\ &KafkaEx.fetch/3) View Source
start_consumers([String.t], (... -> any)) :: {:ok, [pid]}

starts a collection of consumer workers defaulting to KafkaEx.fetch/3 for the message handler

Link to this function validate_uuid(uuid) View Source
validate_uuid(String.t) ::
  {:ok, String.t} |
  {:error, :invalid_uuid}

validates a string as a uuid

Examples

iex> KafkaExHelpers.validate_uuid("33c5aa7e-3f35-47bc-883b-33ea0ace89f0")
{:ok, "33c5aa7e-3f35-47bc-883b-33ea0ace89f0"}

iex> KafkaExHelpers.validate_uuid("some_string")
{:error, :invalid_uuid}