Changelog

View Source
  • 4.5.5
    • Fix infinite MEMBER_ID_REQUIRED rejoin loop against Kafka brokers that take the dynamic-membership path on JoinGroup. Per KIP-394 (Kafka 2.2+, JoinGroup v4+), a member that joins with an empty member_id receives error_code=MEMBER_ID_REQUIRED and a broker-allocated member_id in the same response, and is expected to retry with the new id. brod previously threw the error code before reading the body, dropping the member_id, so every retry sent an empty member_id again and the coordinator died with max_rejoin_attempts (subscribers then restarted with {shutdown, coordinator_failure}, looping forever). Hidden on Kafka 2.3+ because brod's default group_instance_id makes the broker take the KIP-345 static-member fast path; exposed on Kafka 2.2 (no static membership) and on any broker when the user sets {group_instance_id, null} to opt out of static membership. Also fixed: assignments_revoked was called once per stabilize retry instead of once per stabilize cycle, producing duplicate revoke callbacks on any retry path. Added Kafka 2.2 to the CI matrix.
    • Fix deadlock between brod_client' and a per-topicbrod_producers_sup' (issue #662; regression in 4.4.5 / PR #636 widened exposure). The producer supervisor's post_init/1' callback called back intobrod_client' via brod_client:get_partitions_count/2', forming a cycle withbrod_client''s own count_started_children/1' walk insidedo_get_metadata' — under concurrent auto_start_producer' calls the two processes would block on each other indefinitely withtimeout=infinity'. brod_producers_sup:post_init/1' now reads the partition count directly frombrod_client''s workers_tab' ETS cache (anamed_table, protected' table named after the client id, populated by validate_topic_existence/3' immediately before the per-topic supervisor is started). This removes thegen_server:call' back into brod_client' on the hot path and breaks the cycle. The synchronousbrod_client:get_partitions_count/2' is still used as a fallback on cache miss for external callers of brod_producers_sup:start_producer/4'. No public API change. - 4.5.4 - Fix silent fallback inbrod_group_subscriber/v2when Kafka'sOffsetFetchreturnscommitted_offset=-1(no committed offset for the group, e.g. new group, topic recreated,offsets.retention.minutesexpired). Previously the partition was silently dropped from the result and the subscriber started from the consumer's defaultbegin_offset(typicallylatest) with no log. The coordinator now surfaces{begin_offset, per_reset_policy}in#brod_received_assignment{}, and the subscriber resolves it via consumer config:begin_offsetwins if set (backwards compatible), otherwiseoffset_reset_policyis consulted; either fallback is logged. [PR#660](https://github.com/kafka4beam/brod/pull/660), [PR#661](https://github.com/kafka4beam/brod/pull/661). - Fixread_committedisolation:drop_abortedcould stop scanning too early when control records from non-aborted transactions appeared before the abort marker, causing aborted batches to be returned in the record set. [PR#659](https://github.com/kafka4beam/brod/pull/659). - 4.5.3 - Pin kafka_protocol-4.3.4 (crc32cer-1.1.3 and kafka_protocol-4.3.4) If a new re-authentication happens before the connection is still processing requests left-over from the previous re-authentication, the pending requests may get lost. For producers, it means a synced produce call may timeout. For consumers, if a fetch request was lost, it may cause the consumer process to stall indefinitely. - 4.5.2 - Pin kafka_protocol-4.3.2 (crc32cer-1.1.2). - 4.5.1 - Refine warning log forgroup_subscriber_v2when corrdinator exits. - Functionbrod_group_subscriber_v2:health_check/2returnshealthy | rebalancing | {error, [..]}. - 4.5.0 - Addbrod_group_subscriber_v2:health_check/2to check topic-partition consumer connectivity status. - 4.4.7 - Upgrade tokafka_protocol-4.3.0for improved message encoding performance. See [kafka_protocol changelog](https://github.com/kafka4beam/kafka_protocol/blob/master/changelog.md) for details. - Add a warning log when group coordinator 'EXIT' signal is received bygroup_subscriber_v2. - 4.4.6 - Upgrade tokafka_protocol-4.2.8for dependency (crc32cer-1.0.4) to fix its link error. - 4.4.5 - Start supervisor process for the new increase partitions atdo_get_metadatafunction. - Upgrade tokafka_protocol-4.2.7for fast dependency (crc32cer-1.0.3) build. - 4.4.4 - FixedListGroupsAPI request for Kafka Protocol API version 3. - 4.4.3 - Modifybrod_client:get_metadataandbrod_client:get_metadata_safefunction for support multiple topics. - 4.4.2 - Expanded API version ranges to support Kafka 4.0.0 ``` join_group: {0,1} -> {0,6} heartbeat: {0,0} -> {0,4} leave_group: {0,0} -> {0,4} describe_groups: {0,0} -> {0,5} list_groups: {0,0} -> {0,3} create_topics: {0,0} -> {0,4} delete_topics: {0,0} -> {0,4} ``` - 4.4.1 - Upgradekafka_protocolfrom 4.2.3 to 4.2.5 for: -crc32cperformance improvements on ARM. - Replace self messaging ininitwithhandle_continue- 4.4.0 - Supportzstdcompression. - Upgradekafka_protocolfrom 4.1.10 to 4.2.3 for: - logging improvements -crc32cperformance improvements on x86 - 4.3.3 - Catchexitexception when group coordinator calls group member (owner) process. - Addunknown_topic_cache_ttloption in client config. - Doc improvements. - 4.3.2 - Upgradekafka_protocolfrom 4.1.9 to 4.1.10 for partition leader discover/connect timeout fix. - 4.3.1 - Fixedbrod_client:stop_consumerso that it doesn't crash the client process if an unknown consumer is given as argument. - Previously,brod_group_subscriber_v2could leavebrod_consumerprocesses lingering even after its shutdown. Now, those processes are terminated. - 4.3.0 - Split brod-cli out to a separate project [kafka4beam/brod-cli](https://github.com/kafka4beam/brod-cli) - 4.2.0 - Optimize consumer fetch latency. Introduced theshare_leader_connconsumer configuration option (default:false). This setting allows users to opt for the previous behavior if preferred (set totrue). - 4.1.1 - Upgradekafka_protocolfrom version 4.1.5 to 4.1.9. - 4.1.0 - For v2 subscribers, pass offsetack_funas a callback state init parameter. This makes it possible for callback implementation to acknowledge offsets easier. [PR#591](https://github.com/kafka4beam/brod/pull/591) - Type spec fix. [PR#590](https://github.com/kafka4beam/brod/pull/590) - Fixbrod:fold/8to avoid exception when a stable offset points to en empty batch. [PR#589](https://github.com/kafka4beam/brod/pull/589) - 4.0.0 - Removesnappyerfrom default dependency [PR#547](https://github.com/kafka4beam/brod/pull/547). Starting fromkafka_protocol-4.0, it no longer requires compression libraries as rebar dependencies. Nowbrod-4.0has it removed too. For more compression options, see [kafka_protocol/README](https://github.com/kafka4beam/kafka_protocol/blob/master/README.md#compression-support) - 3.19.1 - Made brod-cli to work on OTP 26. [PR#582](https://github.com/kafka4beam/brod/pull/582) ---ssloption is now mandatory if TLS is to be used (previously it can be derived from--cacertfileoption) - TLS version defaults to 1.2, added--ssl-versionsto support explictly setting TLS 1.3 - Support building on OTP 27 [PR#585](https://github.com/kafka4beam/brod/pull/585) Also fixed rebar.lock for dependencykafka_protocol-4.1.5- Guard against crashes inbrod_client:get_consumer/3[PR#587](https://github.com/kafka4beam/brod/pull/587) (originally [PR#581](https://github.com/kafka4beam/brod/pull/581)) - 3.19.0 - Forward unhandled messages in topic/group consumer processes to handle_info/2 callbacks in order to support arbitrary message passing [PR#580](https://github.com/kafka4beam/brod/pull/580) - 3.18.0 - Add transactional APIs. [PR#549](https://github.com/kafka4beam/brod/pull/549) - Fix unnecessary group coordinator restart due tohb_timeoutexception. [PR#578](https://github.com/kafka4beam/brod/pull/578) - Changed supervisor3 progress log level frominfotodebug. [PR#572](https://github.com/kafka4beam/brod/pull/572) - Type spec fix. [PR#571](https://github.com/kafka4beam/brod/pull/571) - Remove unused macro. [PR#575](https://github.com/kafka4beam/brod/pull/575) - 3.17.1 - Upgradekafka_protocolfrom 4.1.3 to 4.1.5 - Allow space after,in comma-separated bootstrapping host:port list - Avoidbadmatchexception when parsing SASL password file - 3.17.0 - Deletedsupervisor3as dependency, module is now moved to brod.git namedbrod_supervisor3. - Upgradesnappyerversion from 1.2.8 to 1.2.9. - Upgradekafka_protocolfrom 4.1.2 to 4.1.3. - In rebar3.config movecoverallsfrompluginstoproject_plugins. - 3.16.8 - Upgradesupervisor3from 1.1.11 to 1.1.12 to log supervisor status atdebuglevel instead ofnoticelevel. - Addbrod_group_subscriber_v2:get_workersfunction to help monitor and check the health of a consumer group. - 3.16.7 - Upgradekafka_protocolfrom 4.1.1 to 4.1.2 to allow handling multiply nested wrapped secrets. - 3.16.6 - Upgradekafka_protocolfrom 4.1.0 to 4.1.1 to support defining SNI for bootstrap hosts without the need to set theverify_peerconfig. - 3.16.5 - Improve the documentation - Addbrod:get_partitions_count_safe/2. It is ensured not to auto-create topics in Kafka even when Kafka has topic auto-creation configured. - Treatoffset = 0commit record asearliest. This is an issue introduced in 3.16.4 which supported subscriber managed offset commits to be represented as{begin_offset, Offset}. The issue is:0is altered by v2 group worker to-1. Handled as offsets greater than 0, v2 group worker assumsbrod_topic_subscriberwould apply+1to it, however-1is actually interpreted aslatest. [#531](https://github.com/kafka4beam/brod/pull/531) fixes it by replacing0withearliest. - 3.16.4 - Allow special begin_offset from consumer managed commits. - Fix specs forbrod_group_subscriber_v2.get_committed_offset- Update kafka_protocol from 4.0.3 to 4.1.0. kafka_protocol 4.1.0 support a different version of the auth plugin interface that also pass the handshake version to the plugin. This is used by the SASL GSSAPI Kerberos plugin brod_gssapi (<https://github.com/kafka4beam/brod_gssapi>) so that the correct handshake protocol is used. The change is backwards compatible as the old interface will be used if the plugin module does not have a function with the name and arity that is required by the new interface. See <https://github.com/kafka4beam/kafka_protocol/pull/102> for details. - 3.16.3 - Fix specs fordelete_topics. - Ensure that partition worker process is alive before returning it. - Update kafka_protocol from 4.0.1 to 4.0.3 in rebar.lock file. - Make consumer 'isolation_level' configurable. - 3.16.2 - Update kafka_protocol from 4.0.1 to 4.0.3. Prior to this change the actual time spent in establishing a Kafka connection might be longer than desired due to the timeout being used in SSL upgrade (if enabled), then API version query. This has been fixed by turning the given timeout config into a deadline, and the sub-steps will try to meet the deadline. see more details here: <https://github.com/kafka4beam/kafka_protocol/pull/92> - Catchtimeoutand otherDOWNreasons when makinggen_servercall tobrod_client,brod_consumerand producer/consumer supervisor, and return asReasonin{error, Reason}. Previously onlynoprocreason is caught. (#492) - Propagateconnect_timeoutconfig tokproAPI functions astimeoutarg affected APIs: connect_group_coordinator, create_topics, delete_topics, resolve_offset, fetch, fold, fetch_committed_offsets (#458) - Fix bad field name in group describe request (#486) - 3.16.1 - Fixbrodscript inbrod-cliin release. - Supportrebalance_timeoutconsumer group option - 3.16.0 - Update to kafka-protocol v4.0.1 - 3.15.6 - Eliminate long-lived anonymous function closures - 3.15.5 - Fix exponential growth ofbrod_producerbuffer - 3.15.4 - Avoid start_link for temp connection usages affected APIs: fetch_committed_offsets, fetch, resolve_offset, create_topics, delete_topics - 3.15.3 - Try to commit acknowledged offsets when brod_group_subscriber_v2 terminates - Fix process leak, close connection after offset is fetched in brod_utils:fetch_committed_offsets/3 - 3.15.2 - Producer: Do not format producer buffer ingen_serverstate - Consumer: Do not commit offsets whenunknown_member_iderror is received - Logging: Changed fromerror_loggertologgermacros - Don't shut down brod_group_subscriber_v2 on previous generation worker termination - Fix brod_group_subscriber_v2 crash on shutdown - Use GitHub Actions instead of Travis CI - Added Elixir example - Rename DEFAULT_TIMEOUT macro in public brod.hrl to BROD_DEFAULT_TIMEOUT. - 3.15.0 (and 3.15.1, for some reason tagged on the same commit) - Fix unknown types - Build on OTP-23 - 3.14.0 NOTE: This release changes internal states of brod worker processes in a way that cannot by applied on a running system. brod application and all brod workers should be shot down for the time of the upgrade. - Introduced a new optional terminate/2 callback to brod_topic_subscriber and brod_group_subscriber_v2 behaviors - Introduced new API functions for starting topic subscribers: brod_topic_subscriber:start_link/1 and brod:start_link_topic_subscriber/1 Old APIs - brod:start_link_topic_subscriber/5, - brod:start_link_topic_subscriber/6 - brod:start_link_topic_subscriber/7 - brod_topic_subscriber:start_link/6 - brod_topic_subscriber:start_link/7 - brod_topic_subscriber:start_link/8 are deprecated and will be removed in the next major release - 3.13.0 - Update supervisor3 dependency to 1.1.11 - brod_group_subscriber_v2 behavior handles worker crashes - Makefile fix for hex publishing - Reverse Changelog order - Fix typos and dialyzer warnings - 3.12.0 - Enable passing custom timeout to resolve_offset - 3.11.0 - Improve compatibility with EventHub - 3.10.0 - Stop supporting erlang.mk - Stop supporting rebar - Update kafka_protocol dependency to 2.3.6 - Add new brod_group_coordinator:update_topics API - 3.9.5 - Bump kafka_protocol dependency to 2.3.3 - 3.9.4 - Handle undefined fetcher in fold loop exception - 3.9.3 - Remove vsn-check dependency from default Makefile target This enables using brod with erlang.mk + hex - Bump kafka_protocol dependency to 2.3.2 - 3.9.2 - Fix corrupted package published to hex - 3.9.1 - Receive pending acks after assignments_revoked is invoked - 3.9.0 - Updated kafka_protocol dependency to 2.3.1 - (Experimental) Added group_subscriber_v2 behavior - Added API for topic deletion and creation - 3.8.1 - Handle the case when high_watermark < last_stable_offset in fetch resp - 3.8.0 - Bump to kafka_protocol 2.2.9 (allowatom()hostname) - Addbrod:fold/8. This API spawns a process to fetch-ahead while folding the previously fetched batch.brod-cli'sfetchcommand is updated to call thisfoldAPI for better performance. - Add callbacks to allowbrod_client:stop_producerandbrod_client:stop_consumerto remove the stopped child references from the supervisor and clean up the client ets table to allow later restart. - Support scram SASL authentication in brod-cli - Made possible to start-link or supervisebrod_consumerin user apps, instead of always underbrod_client'sbrod_consumers_sup- 3.7.11 - Fix a bug when dropping aborted transactions for compacted topics - 3.7.10 - Compare begin_offset with last stable offset before advancing to next offset in case empty batch is received. Prior to this version, fetch attempts on unstable messages (messages belong to open transactions (transactions which are neither committed nor aborted), may result in an empty message set, thenbrod_consumerorbrod_utils:fetchjumps to the next offset (if it is less than high-watermark offset). - 3.7.9 - Fix brod-cli escript include apps - Fix brod-cli sub-record formatting crash - Upgrade to kafka_protocol 2.2.8 to discard replica_not_available error code in metadata response - Fix empty responses field in fetch response #323 - 3.7.8 - Drop batches in aborted transactions (and all control batches) also improve offset fast-forwarding when empty batches are received - 3.7.7 - Fixbadrecordrace: message-set is delivered tobrod_group_subscriberafter unsubscribed frombrod_consumer. - 3.7.6 - Fix produce message input type spec to make it backward compatible (bug introduced in 3.7.3) - 3.7.5 - Bump kafka_protocol version to 2.2.7 - Fix empty assignment handling. In case a group member has no partition assigned,member_assignmentdata field in group sync response can either benull(kafka 0.10) or a struct having emptytopic_partitions(kafka 0.11 or later). The later case was not handled properly inbrodbefore this fix. - 3.7.4 - Add callback to make user_data in group join request - 3.7.3 - Bump kafka_protocol version to 2.2.3 - Discard stale async-ack messages to group subscriber - 3.7.2 - Pr #298: Subscriber now automatically reconnects to Kafka on topic rebalances where the previous partition leader no longer holds the partition at all. - Pr #299: Fix topic subscriber returning partition offsets from callback module's init. - 3.7.1 - Fix brod_topic_subscriber and brod_group_subscriber re-subscribe behaviour to avoid fetching duplicated messages. - Add 'random' and 'hash' partitioner for produce APIs - Allowbrod_group_subscrber:assign_partitions/3to return an updated cb_state -brod_client:get_connectionis back - Make it possible to run tests on both mac and linux - Position group leader member at the head of members list when assigning partitions - 3.7.0 - Addbrod_group_subscriber:ack/5andbrod_group_subscriber:commit/4to let group subscribers commit offsets asynchronously - Pr #284: In compacted topics, Kafka may return a MessageSet which contains only messages before the desired offset. Just keep reading forward in this case. - Issue #285brod_consumerno longer restart onleader_not_availbleandnot_leader_for_partitionerror codes received in fetch response. It resets connection and rediscover leader after delay. - 3.6.2 - Allowbrod_topic_subscriberto explicitly start consuming from partition offset 0 (by passing in a committed offset of -1). - 3.6.1 - Make produce request version configurable asproduce_req_vsninbrod:producer_config()- Upgradekafka_protocolto2.1.2to support alpine/busybox build - 3.6.0 - Moved 3 modules to kafka_protocol: -brod_sock->kpro_connection-brod_auth_backed->kpro_auth_backend-brod_kafka_requests->kpro_sent_reqs-#kafka_message.keyand#kafka_message.valueare now alwaysbinary()(they were of specundefined | binary()prior to this version). i.e. empty bytes are now decoded as<<>>instead ofundefined. This may cause dialyzer check failures. -brod_clientno longer logs about metadata socket down, it had been confusing rather than being helpful - There is no more cool-down delay for metadata socket re-establishment -brod_group_coordinatordefault session timeout changed from 10 seconds to 30, and heartbeat interval changed from 2 seconds to 5. - Addbrod:produce_cb/4andbrod:produce_cb/6to support user defined callback as produce ack handler. - Addbrod:produce_no_ack/3andbrod:produce_no_ack/5. -min_compression_batch_sizeis removed from producer config. - Support magic v2 batch/message format (message headers). - Use rebar3 as primary build tool for travis-ci, erlang.mk is still supported. - SupportSCRAM-SHA-256andSCRAM-SHA-512SASL authentication mechanisms. - 3.5.2 - Fix issue #263: Kafka 0.11 may send empty batch in fetch response when messages are deleted in compacted topics. - 3.5.1 - Addextra_sock_optsclient socket options. It would be helpful for tuning the performance of tcp socket. - 3.5.0 - Add*_offsetvariants toproduceAPIs, returning the base offsets that were assigned by Kafka. Producers need to be restarted when upgrading to this version. - 3.4.0 - Addprefetch_bytesconsumer config.brod_consumershould stop fetch-ahead only when bothprefetch_countandprefetch_byteslimits are exceeded - 3.3.5 - Fix issue #252 -- Kafka 0.11 and 1.0 have more strict validations on compressed batch wrapper message. Changed kafka_protocol 1.1.2 has the wrapper timestamp and offsets fixed. - 3.3.4 - Fix issue #247 -- revert the handling of offset = -1 in offset_fetch_response, bug introduced in 3.3.0 offset = -1 in offset_fetch_response is an indicator of 'no commit' should be ignored (not taken as 'latest') - 3.3.3 - Add a --no-api-vsn-query option for brod-cli to support kafka 0.9 - Bump kafka_protocol to 1.1.1 to fix relative offsets issue so brod-cli can fetch compressed batches as expected, also brod_consumer can start picking fetch request version - Upgrade roundrobin group protocol to roundrobin_v2 to fix offset commit incompatiblility with kafka spec and monitoring tools etc. see <https://github.com/klarna/brod/issues/241> for details - 3.3.2 - Enhancements - Add sh script to wrap brod-cli escript for erts dir auto discover - Detailed log for connection estabilishment failures - Bug Fixes - Demonitor producer pid after sync_produce_request timeout - 3.3.1 - Fix brod-cli commits command redandunt socket usage. - 3.3.0 - New Features - Support offset_fetch_request version 1 - 2 - Provide APIs to reset committed offsets - Support offset commit in brod-cli - Improved group coordinator logging: 1) stop showing member ID, 2) show callback process. - Cache queried version ranges per kafka host - Brod-cli 'offset' command by default resolves offsets for all partitions if '--partition' option is 'all' or missing - Enhancements - Brod rock with elvis with travis - Travis run dialyzer check - Bug Fixes - Fixed brod-cli offset_fetch_request version when working with kafka 0.10.1.x or earlier - Make group coordinator restart on heartbeat timeout - 3.2.0 - New Features - Support produce_request version 0 - 2 - Support fetch_request version 0 - 3 - Support offsets_request version 0 - 1 - Support metadata_request version 0 - 2 - Support message create time by allowingValueto be[{Ts, Key, Value}]inbrod:produceandbrod:produce_syncAPIs - Bind all#kafka_message{}fields to variables so they can be used in brod-clifetchcommand's--fmtexpression - Bug Fixes - Remove sasl application from dependent app list - 3.1.0 - New Features - New message type option for topic and group subscribers that specifies whether to handle a Kafkamessageormessage_set. - 3.0.0 - New Features - New APIbrod:connect_group_cordinatorto establish a sockt towards group coordinator broker. - New APIbrod:fetch_committed_offsetsto fetch consumer group committed offsets. - New APIbrod:list_groups,brod:list_all_groupsandbrod:describe_groups/3. - Brod-cli new commandgroupsto list / describe consumer groups. - Brod-cli new commandcommitsto list comsuer group committed offsets. - Backward-incompatible changes -brod:get_offsetsAPI replaced withbrod:resolve_offset. Reason:brod:get_offsetsandbrod_utils:fetch_offsetsare very confusing, because they look like fetching consumer group committed offsets. Also, the return value has been changed from a list of offsets to a single offset. -brod:get_metadatareturn value changed from#kpro_MetadataResponse{}record tokpro:struct(). -brod_utils:fetch/4is removed, usemake_fetch_fun/8instead. ---countoption removed frombrod-clioffsetcommand, The command now resolves only one (or maybe none) offset at a time. - All type specs defined inbrod.hrlare moved to exported types frombrod.erl- Backward-compatible changes -#kafka_message{}record is extended with new fieldsts_typeandts. -#kafka_message.crcchanged from signed to unsigned integer. - 2.5.1 - Fix ignored commit history when committed offset is 0 (bug) - 2.5.0 - Pluggable SASL authentication backend (contributor: ElMaxo) - Brod-cli support extra ebin to code path - Fix group subscriber duplicated loopback messages (bug) - SASL-PLAIN username password in text file - Hide SASL-PLAIN password in an anonymous function inbrod_client` state - 2.4.1 Fixed brod-cli typo fix - 2.4.0 Added brod-cli