Module brod_utils

Data Types

conn_config()

conn_config() = brod:conn_config()

connection()

connection() = kpro:connection()

endpoint()

endpoint() = brod:endpoint()

fetch_fun()

fetch_fun() = fun((offset()) -> {ok, {offset(), [brod:message()]}} | {error, any()})

group_id()

group_id() = brod:group_id()

offset()

offset() = brod:offset()

offset_time()

offset_time() = brod:offset_time()

partition()

partition() = brod:partition()

req_fun()

req_fun() = fun((offset(), kpro:count()) -> kpro:req())

topic()

topic() = brod:topic()

Function Index

assert_client/1Assert client_id is an atom().
assert_group_id/1Assert group_id is a binary().
assert_topic/1Assert topic is a binary().
assert_topics/1Assert a list of topic names [binary()].
bytes/1Return message set size in number of bytes.
describe_groups/3Send describe_groups_request and wait for describe_groups_response.
epoch_ms/0Milliseconds since beginning of the epoch (midnight Jan 1, 1970 (UTC)).
fetch/4Fetch a message-set.
fetch/5Fetch a single message set from the given topic-partition.
fetch_committed_offsets/3Fetch commited offsets for the given topics in a consumer group.
fetch_committed_offsets/4Fetch committed offsets for the given topics in a consumer group.
flatten_batches/3Make a flat message list from decoded batch list.
fold/8
get_metadata/1Try to connect to any of the bootstrap nodes and fetch metadata for all topics.
get_metadata/2Try to connect to any of the bootstrap nodes and fetch metadata for the given topics.
get_metadata/3Try to connect to any of the bootstrap nodes using the given connection options and fetch metadata for the given topics.
get_stable_offset/1last_stable_offset is added in fetch response version 4 This function takes high watermark offset as last_stable_offset in case it's missing.
group_per_key/1Group values per-key in a key-value list.
group_per_key/2Group values per-key for the map result of a list.
init_sasl_opt/1Hide sasl plain password in an anonymous function to avoid the plain text being dumped to crash logs.
is_normal_reason/1Check terminate reason for a gen_server implementation.
is_pid_alive/1
list_all_groups/2List all groups in the given cluster.
list_groups/2List all groups in the given coordinator broker.
log/3simple wrapper around error_logger.
make_batch_input/2Make batch input for kafka_protocol.
make_fetch_fun/4Make a fetch function which should expand max_bytes when it is not big enough to fetch one signle message.
make_part_fun/1
os_time_utc_str/0Get now timestamp, and format as UTC string.
parse_rsp/1Parse decoded kafka response (#kpro_rsp{}) into a more generic representation.
request_sync/2
request_sync/3
resolve_offset/4Resolve timestamp or semantic offset to real offset.
resolve_offset/5Resolve timestamp to real offset.

Function Details

assert_client/1

assert_client(Client::brod:client_id() | pid()) -> ok | no_return()

Assert client_id is an atom().

assert_group_id/1

assert_group_id(GroupId::group_id()) -> ok | no_return()

Assert group_id is a binary().

assert_topic/1

assert_topic(Topic::topic()) -> ok | no_return()

Assert topic is a binary().

assert_topics/1

assert_topics(Topics::[topic()]) -> ok | no_return()

Assert a list of topic names [binary()].

bytes/1

bytes(Msgs::bord:batch_input()) -> non_neg_integer()

Return message set size in number of bytes. NOTE: This does not include the overheads of encoding protocol. such as magic bytes, attributes, and length tags etc.

describe_groups/3

describe_groups(CoordinatorEndpoint::endpoint(), ConnCfg::conn_config(), IDs::[brod:group_id()]) -> {ok, kpro:struct()} | {error, any()}

Send describe_groups_request and wait for describe_groups_response.

epoch_ms/0

epoch_ms() -> kpro:msg_ts()

Milliseconds since beginning of the epoch (midnight Jan 1, 1970 (UTC)).

fetch/4

fetch(Conn::connection(), ReqFun::req_fun(), Offset::offset(), MaxBytes::kpro:count()) -> {ok, {offset(), [brod:message()]}} | {error, any()}

Fetch a message-set. If the given MaxBytes is not enough to fetch a single message, expand it to fetch exactly one message

fetch/5

fetch(Hosts::connection() | brod:client_id() | brod:bootstrap(), Topic::topic(), Partition::partition(), Offset::offset(), Opts::brod:fetch_opts()) -> {ok, {offset(), [brod:message()]}} | {error, any()}

Fetch a single message set from the given topic-partition.

fetch_committed_offsets/3

fetch_committed_offsets(Client::brod:client(), GroupId::group_id(), Topics::[topic()]) -> {ok, [kpro:struct()]} | {error, any()}

Fetch commited offsets for the given topics in a consumer group. 1. Get broker endpoint by calling brod_client:get_group_coordinator 2. Establish a connecton to the discovered endpoint. 3. send offset_fetch request and wait for response. If Topics is an empty list, fetch offsets for all topics in the group

fetch_committed_offsets/4

fetch_committed_offsets(BootstrapEndpoints::[endpoint()], ConnCfg::conn_config(), GroupId::group_id(), Topics::[topic()]) -> {ok, [kpro:struct()]} | {error, any()}

Fetch committed offsets for the given topics in a consumer group. 1. try find out the group coordinator broker from the bootstrap hosts 2. send offset_fetch request and wait for response. If Topics is an empty list, fetch offsets for all topics in the group

flatten_batches/3

flatten_batches(BeginOffset::offset(), Header::map(), Batches0::[kpro:batch()]) -> {offset(), [kpro:message()]}

Make a flat message list from decoded batch list. Return the next beging-offset together with the messages.

fold/8

fold(Hosts::connection() | brod:client_id() | brod:bootstrap(), Topic::topic(), Partition::partition(), Offset::offset(), Opts::brod:fetch_opts(), Acc, Fun::brod:fold_fun(Acc), Limits::brod:fold_limits()) -> brod:fold_result()

get_metadata/1

get_metadata(Hosts::[endpoint()]) -> {ok, kpro:struct()} | {error, any()}

Try to connect to any of the bootstrap nodes and fetch metadata for all topics

get_metadata/2

get_metadata(Hosts::[endpoint()], Topics::all | [topic()]) -> {ok, kpro:struct()} | {error, any()}

Try to connect to any of the bootstrap nodes and fetch metadata for the given topics

get_metadata/3

get_metadata(Hosts::[endpoint()], Topics::all | [topic()], ConnCfg::conn_config()) -> {ok, kpro:struct()} | {error, any()}

Try to connect to any of the bootstrap nodes using the given connection options and fetch metadata for the given topics.

get_stable_offset/1

get_stable_offset(Header) -> any()

last_stable_offset is added in fetch response version 4 This function takes high watermark offset as last_stable_offset in case it's missing.

group_per_key/1

group_per_key(List::[{Key, Val}]) -> [{Key, [Val]}]

Group values per-key in a key-value list.

group_per_key/2

group_per_key(MapFun::fun((term()) -> {Key, Val}), List::[term()]) -> [{Key, [Val]}]

Group values per-key for the map result of a list.

init_sasl_opt/1

init_sasl_opt(Config::brod:client_config()) -> brod:client_config()

Hide sasl plain password in an anonymous function to avoid the plain text being dumped to crash logs

is_normal_reason/1

is_normal_reason(X1) -> any()

Check terminate reason for a gen_server implementation

is_pid_alive/1

is_pid_alive(Pid) -> any()

list_all_groups/2

list_all_groups(Endpoints::[endpoint()], Options::conn_config()) -> [{endpoint(), [brod:cg()] | {error, any()}}]

List all groups in the given cluster. NOTE: Exception if failed against any of the coordinator brokers.

list_groups/2

list_groups(Endpoint::endpoint(), ConnCfg::conn_config()) -> {ok, [brod:cg()]} | {error, any()}

List all groups in the given coordinator broker.

log/3

log(X1::info | warning | error, Fmt::string(), Args::[any()]) -> ok

simple wrapper around error_logger. NOTE: keep making MFA calls to error_logger to 1. allow logging libraries such as larger parse_transform 2. be more xref friendly

make_batch_input/2

make_batch_input(Key::brod:key(), Value::brod:value()) -> brod:batch_input()

Make batch input for kafka_protocol.

make_fetch_fun/4

make_fetch_fun(Conn::pid(), Topic::topic(), Partition::partition(), FetchOpts::brod:fetch_opts()) -> fetch_fun()

Make a fetch function which should expand max_bytes when it is not big enough to fetch one signle message.

make_part_fun/1

make_part_fun(F::brod:partitioner()) -> brod:partition_fun()

os_time_utc_str/0

os_time_utc_str() -> string()

Get now timestamp, and format as UTC string.

parse_rsp/1

parse_rsp(Kpro_rsp::kpro:rsp()) -> ok | {ok, term()} | {error, any()}

Parse decoded kafka response (#kpro_rsp{}) into a more generic representation. Return ok if it is a trivial 'ok or not' response without data fields Return {ok, Result} for some of the APIs when no error-code found in response. Result could be a transformed representation of response message body #kpro_rsp.msg or the response body itself. For some APIs, it returns {error, CodeOrMessage} when error-code is not no_error in the message body. NOTE: Not all error codes are interpreted as {error, CodeOrMessage} tuple. for some of the complex response bodies, error-codes are retained for caller to parse.

request_sync/2

request_sync(Conn::connection(), Req::kpro:req()) -> ok | {ok, term()} | {error, any()}

request_sync/3

request_sync(Conn::connection(), Kpro_req::kpro:req(), Timeout::infinity | timeout()) -> ok | {ok, term()} | {error, any()}

resolve_offset/4

resolve_offset(Pid::pid(), Topic::topic(), Partition::partition(), Time::offset_time()) -> {ok, offset()} | {error, any()}

Resolve timestamp or semantic offset to real offset. The give pid should be the connection to partition leader broker.

resolve_offset/5

resolve_offset(Hosts::[endpoint()], Topic::topic(), Partition::partition(), Time::offset_time(), ConnCfg::conn_config()) -> {ok, offset()} | {error, any()}

Resolve timestamp to real offset.


Generated by EDoc