时间戳

首页 > 解决方案 > 卡夫卡抛出“org.apache.kafka.clients.consumer.CommitFailedException”

java - 卡夫卡抛出“org.apache.kafka.clients.consumer.CommitFailedException”

问题描述

我已经使用spring-kafka库开发了 Kafka 消费者应用程序，并使用了手动提交的默认消费者配置。

我正在运行两个应用程序实例，监听两个不同的 Kafka 主题。在执行负载测试时，我观察到我在只有一个应用程序中遇到更高负载的错误：

Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. 

    This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, 
    which typically implies that the poll loop is spending too much time message processing. 

    You can address this either by increasing the session timeout or by reducing the maximum size of batches 
    returned in poll() with max.poll.records.


org.apache.kafka.clients.consumer.CommitFailedException: 
    Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. 

    This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, 
    which typically implies that the poll loop is spending too much time message processing. 


You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records.
    \n org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.sendOffsetCommitRequest(ConsumerCoordinator.java:725)

我阅读了几篇文章，发现如果消费者花费大量时间处理消息并且代理没有获取有关消费者活跃度的信息，那么就会发生消费者重新平衡，并且对于未提交的消息将引发上述异常。

我通过将 max.poll.interval.ms 设置为 INEGER.MAX_VALUE 解决了上述错误。但我想知道为什么我只在一个实例中超过错误，为什么其他实例在更高负载下按预期工作。

谁能分享max.poll.interval.ms这个问题的正确根本原因和理想价值或适当的解决方案

标签： javaapache-kafkakafka-consumer-apispring-kafka

解决方案

造成这种情况的一个原因可能是您poll()接收了很多消息，这就是为什么需要花费大量时间来处理所有这些消息。
max.poll.records定义了在一次调用中返回的最大记录数poll()。
根据kafka 文档，默认值为 500。
您可以尝试将其设置为更小的值，看看是否能解决您的问题。

推荐阅读