时间戳

首页 > 解决方案 > Karafka start_from_beginning not working as expected

ruby - Karafka start_from_beginning not working as expected

问题描述

I've created a project to help with my understanding of Kafka. It's set up as three identical Rails apps all inside Docker with Karafka configured to consume the messages - if you create a record in one, it's replicated across to the other two. I assumed that the start_from_beginning setting would mean that every time the Karafka server was restarted it would start from offset 0, but that does seem to be the case. Can someone please explain what I've done wrong or correct my understanding.

Here are the two significant sections from karafka.rb

  setup do |config|
    config.kafka.seed_brokers = %w[kafka://kafka:9092]
    config.client_id = "app_#{ ENV['APP_ID'] }"
    config.logger = Rails.logger
  end

...

  consumer_groups.draw do
    topic :party do
      consumer PartyConsumer
      start_from_beginning true
    end
  end

I have already tried putting config.kafka.start_from_beginning = true in the config section of karafka.rb but no joy.

When I create a record in one of the apps, it's sync over to the other two. This is what I was trying to do:

At this point, I was expecting the database to be re-created from Kafka by rewinding to offset 0 and replaying all the messages. What have I missed?

The full project is here: https://github.com/jcleary/kafka-demo

标签： rubyapache-kafkakarafka

解决方案

问题实际上是我对 的理解start_from_beginning，其目的是决定新消费者的行为方式。现有的消费者总是被期望从他们离开的地方重新开始。

我找到了两种方法来实现我正在寻找的东西：

1 - 使用 Kafka 的 seek 功能（推荐）：

来自维基：https ://github.com/karafka/karafka/wiki/Events-monitoring-and-logging#using-the-connectionlistenerbefore_fetch_loop-for-topic-seeking

2 - 改变client_idin karafka.rb，例如

class KarafkaApp < Karafka::App
  setup do |config|
    config.kafka.seed_brokers = %w[kafka://kafka:9092]
    config.client_id = "app_#{ ENV['APP_ID'] }-#{ SecureRandom.uuid }"
    config.logger = Rails.logger
  end
  ...

使用SecureRandom.uuid将确保您的消费者具有唯一client_id性，因此重新处理所有消息。

选项 2 更像是一种 hack，但根据您的用例，可能是您正在寻找的。

推荐阅读