Module brod_consumer

Behaviours: gen_server.

Data Types

config()

config() = proplists:proplist()

options()

options() = brod:consumer_options()

partition()

partition() = brod:partition()

topic()

topic() = brod:topic()

Function Index

ack/2Subscriber confirms that a message (identified by offset) has been consumed, consumer process now may continue to fetch more messages.
code_change/3
debug/2Enable/disable debugging on the consumer process.
get_connection/1Get connection pid.
handle_call/3
handle_cast/2
handle_info/2
init/1
start_link/4Equivalent to start_link(ClientPid, Topic, Partition, Config, []).
start_link/5Start (link) a partition consumer.
stop/1
stop_maybe_kill/2
subscribe/3Subscribe or resubscribe on messages from a partition.
terminate/2
unsubscribe/2Unsubscribe the current subscriber.

Function Details

ack/2

ack(Pid::pid(), Offset::brod:offset()) -> ok

Subscriber confirms that a message (identified by offset) has been consumed, consumer process now may continue to fetch more messages.

code_change/3

code_change(OldVsn, State, Extra) -> any()

debug/2

debug(Pid::pid(), File::print | string() | none) -> ok

Enable/disable debugging on the consumer process. debug(Pid, print) prints debug info on stdout debug(Pid, File) prints debug info into a File

get_connection/1

get_connection(Pid) -> any()

Get connection pid. Test/debug only.

handle_call/3

handle_call(Call, From, State) -> any()

handle_cast/2

handle_cast(Cast, State) -> any()

handle_info/2

handle_info(Info, State) -> any()

init/1

init(X1) -> any()

start_link/4

start_link(Bootstrap::pid() | brod:bootstrap(), Topic::topic(), Partition::partition(), Config::config()) -> {ok, pid()} | {error, any()}

Equivalent to start_link(ClientPid, Topic, Partition, Config, []).

start_link/5

start_link(Bootstrap::pid() | brod:bootstrap(), Topic::topic(), Partition::partition(), Config::config(), Debug::[any()]) -> {ok, pid()} | {error, any()}

Start (link) a partition consumer. Possible configs: min_bytes (optional default = 0): Minimal bytes to fetch in a batch of messages max_bytes (optional default = 1MB): Maximum bytes to fetch in a batch of messages NOTE: this value might be expanded to retry when it is not enough to fetch even one single message, then slowly shrinked back to this given value. max_wait_time (optional, default = 10000 ms): Max number of seconds allowd for the broker to collect min_bytes of messages in fetch response sleep_timeout (optional, default = 1000 ms): Allow consumer process to sleep this amout of ms if kafka replied 'empty' message-set. prefetch_count (optional, default = 10): The window size (number of messages) allowed to fetch-ahead. prefetch_bytes (optional, default = 100KB): The total number of bytes allowed to fetch-ahead. brod_consumer is greed, it only stops fetching more messages in when number of unacked messages has exceeded prefetch_count AND the unacked total volume has exceeded prefetch_bytes begin_offset (optional, default = latest): The offset from which to begin fetch requests. offset_reset_policy (optional, default = reset_by_subscriber) How to reset begin_offset if OffsetOutOfRange exception is received. reset_by_subscriber: consumer is suspended (is_suspended=true in state) and wait for subscriber to re-subscribe with a new 'begin_offset' option. reset_to_earliest: consume from the earliest offset. reset_to_latest: consume from the last available offset. size_stat_window: (optional, default = 5) The moving-average window size to caculate average message size. Average message size is used to shrink max_bytes in fetch requests after it has been expanded to fetch a large message. Use 0 to immediately shrink back to original max_bytes from config. A size esitmation allows users to set a relatively small max_bytes, then let it dynamically adjust to a number around PrefetchCount * AverageSize

stop/1

stop(Pid::pid()) -> ok | {error, any()}

stop_maybe_kill/2

stop_maybe_kill(Pid::pid(), Timeout::timeout()) -> ok

subscribe/3

subscribe(Pid::pid(), SubscriberPid::pid(), ConsumerOptions::options()) -> ok | {error, any()}

Subscribe or resubscribe on messages from a partition. Caller may pass in a set of options which is an extention of consumer config to update the parameters such as max_bytes and max_wait_time etc. also to update the start point (begin_offset) of the data stream. Possible options: all consumer configs as documented for start_link/5 begin_offset (optional, default = latest) A subscriber may consume and process messages then persist the associated offset to a persistent storage, then start (or restart) with last_processed_offset + 1 as the begin_offset to proceed. By default, it fetches from the latest available offset.

terminate/2

terminate(Reason, State) -> any()

unsubscribe/2

unsubscribe(Pid::pid(), SubscriberPid::pid()) -> ok | {error, any()}

Unsubscribe the current subscriber.


Generated by EDoc