apache-kafka - 使用 docker-compose 在多个节点上的 Kafka 集群

Question

我正在尝试使用 docker-compose 在 3 个节点上设置 Kafka 集群。

1. 节点：

version: '3.1'

services:
  zookeeper:
    image: zookeeper:3.7.0
    container_name: zookeeper
    hostname: zookeeper
    ports:
      - 2181:2181
      - 2888:2888
      - 3888:3888
    environment:
      ZOO_MY_ID: 1
      ZOO_SERVERS: server.1=0.0.0.0:2888:3888;2181 server.2=node2:2888:3888;2181 server.3=node3:2888:3888;2181

  kafka:
    image: wurstmeister/kafka:2.13-2.7.0
    container_name: kafka
    hostname: kafka
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: node1:2181,node2:2181,node3:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'
      KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,OUTSIDE://node1:9095
      KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9095
      KAFKA_CREATE_TOPICS: "test:4:1"
    ports:
      - 9095:9095

2. 节点：

version: '3.1'

services:
  zookeeper:
    image: zookeeper:3.7.0
    container_name: zookeeper
    hostname: zookeeper
    ports:
      - 2181:2181
      - 2888:2888
      - 3888:3888
    environment:
      ZOO_MY_ID: 2
      ZOO_SERVERS: server.1=node1:2888:3888;2181 server.2=0.0.0.0:2888:3888;2181 server.3=node3:2888:3888;2181

  kafka:
    image: wurstmeister/kafka:2.13-2.7.0
    container_name: kafka
    hostname: kafka
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 2
      KAFKA_ZOOKEEPER_CONNECT: node1:2181,node2:2181,node3:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'
      KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,OUTSIDE://node2:9095
      KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9095
      #KAFKA_CREATE_TOPICS: "test:4:1"
    ports:
      - 9095:9095

3. 节点：

version: '3.1'

services:
  zookeeper:
    image: zookeeper:3.7.0
    container_name: zookeeper
    hostname: zookeeper
    ports:
      - 2181:2181
      - 2888:2888
      - 3888:3888
    environment:
      ZOO_MY_ID: 3
      ZOO_SERVERS: server.1=node1:2888:3888;2181 server.2=node2:2888:3888;2181 server.3=0.0.0.0:2888:3888;2181

  kafka:
    image: wurstmeister/kafka:2.13-2.7.0
    container_name: kafka
    hostname: kafka
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 3
      KAFKA_ZOOKEEPER_CONNECT: node1:2181,node2:2181,node3:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'
      KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,OUTSIDE://node3:9095
      KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9095
      #KAFKA_CREATE_TOPICS: "test:4:1"
    ports:
      - 9095:9095

当我运行它时（在各个节点上）：

docker-compose -f docker-compose-node1.yml -d
docker-compose -f docker-compose-node2.yml -d
docker-compose -f docker-compose-node3.yml -d

如果我检查日志，则没有错误。相反，我可以看到类似的东西：

INFO Registered broker 1 at path /brokers/ids/1 ...
INFO Registered broker 2 at path /brokers/ids/2 ...
INFO Registered broker 3 at path /brokers/ids/3 ...

有两件事很奇怪，首先在我尝试创建主题的第一个节点上，日志停在

creating topics: test:4:1
Created topic test.

如果我设置了一个节点（本地主机），一切都设置得很好。第二件事是node2总是被选为控制器。这就是我用 kafkacat 得到的：

kafkacat -b node1:9095,node2:9095,node3:9095 -L
Metadata for all topics (from broker 2: node2:9095/2):
 3 brokers:
  broker 2 at node2:9095 (controller)
  broker 3 at node3:9095
  broker 1 at node1:9095
 1 topics:
  topic "test" with 4 partitions:
    partition 0, leader 2, replicas: 2, isrs: 2
    partition 1, leader 3, replicas: 3, isrs: 3
    partition 2, leader 1, replicas: 1, isrs: 1
    partition 3, leader 2, replicas: 2, isrs: 2

但是如果我运行：

docker exec -it kafka kafka-console-consumer.sh --bootstrap-server node1:9095,node2:9095,node3:9095 --topic test --from-beginning

我收到大量警告：

[2021-04-12 14:31:06,848] WARN [Consumer clientId=consumer-console-consumer-95791-1, groupId=console-consumer-95791] Received unknown topic or partition error in ListOffset request for partition test-2 (org.apache.kafka.clients.consumer.internals.Fetcher)

和

[2021-04-12 14:31:06,848] WARN [Consumer clientId=consumer-console-consumer-95791-1, groupId=console-consumer-95791] Received unknown topic or partition error in ListOffset request for partition test-1 (org.apache.kafka.clients.consumer.internals.Fetcher)

就像没有分区 1 和 2 一样。

如果我尝试产生一条消息：

docker exec -it kafka-tn kafka-console-producer.sh --bootstrap-server tn00:9095,tn01:9095,tn02:9095 --topic test

我收到很多错误：

[2021-04-12 14:40:33,273] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 41 : {test=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)

我在这里做了一些疯狂的事情还是只是一些小的错误配置？

标签： apache-kafkadocker-composeapache-zookeeper