View Source Publisher Example
Ensure :brod
is added to your deps on mix.exs
defp deps do
[
{:brod, "~> 3.10.0"}
]
end
client-configuration
Client Configuration
Adding the following configuration (e.g. config/dev.exs
)
import Config
config :brod,
clients: [
kafka_client: [
endpoints: [localhost: 9092],
auto_start_producers: true,
# The following :ssl and :sasl configs are not
# required when running kafka locally unauthenticated
ssl: true,
sasl: {
:plain,
System.get_env("KAFKA_CLUSTER_API_KEY"),
System.get_env("KAFKA_CLUSTER_API_SECRET")
}
]
]
Note: kafka_client
can be any valid atom. And :endpoints
accepts multiple host port tuples (e.g. endpoints: [{"192.168.0.2", 9092}, {"192.168.0.3", 9092}, ...]
).
publisher
Publisher
To send a message with brod we can use the :brod.produce_sync/5
function
defmodule BrodExample.Publisher do
def publish(topic, partition, key, message) do
:brod.produce_sync(:kafka_client, topic, :hash, key, message)
end
end
using-partition-key
Using partition key
When providing :hash
as the partition when calling :brod.produce_sync/5
is equivalent to the following:
{:ok, count} = :brod.get_partitions_count(:kafka_client, topic)
:brod.produce_sync(:kafka_client, topic, :erlang.phash2(key, count), key, message)
Internally brod will get the partition count, generate a hash for the key within the range of partitions, and publish the message to the calculated hash. This is the same sticky routing that Kafka's ProducerRecord implements:
If no partition is specified but a key is present a partition will be chosen using a hash of the key. If neither key nor partition is present a partition will be assigned in a round-robin fashion.