docker - 无法将第二个 kafka 实例 dockerize 到 zookeeper
问题描述
我想用两个 kafka 实例对一个 kafka 集群进行 dockerize。
运行动物园管理员
docker run -it --name zookeeper -p 2181:2181 -p 2888:2888 -p 3888:3888 debezium/zookeeper
运行卡夫卡
docker run -it --name kafka -p 9092:9092 -e ADVERTISED_HOST_NAME=$(hostname -f) --link zookeeper:zookeeper debezium/kafka
运行一个 zookeeper 和一个 kafka 是成功的,但是添加了第二个 kafka 容器,如下所示
docker run -it --name kafka2 -p 9096:9092 -e ADVERTISED_HOST_NAME=$(hostname -f) --link zookeeper:zookeeper debezium/kafka
给我以下错误:
2019-09-24 04:11:48,728 - ERROR [main:Logging@74] - Error while creating ephemeral at /brokers/ids/1, node already exists and owner '72057611679825940' does not match current session '72057611679825967'
2019-09-24 04:11:48,748 - ERROR [main:MarkerIgnoringBase@159] - [KafkaServer id=1] Fatal error during KafkaServer startup. Prepare to shutdown
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists
at org.apache.zookeeper.KeeperException.create(KeeperException.java:122)
at kafka.zk.KafkaZkClient$CheckedEphemeral.getAfterNodeExists(KafkaZkClient.scala:1784)
at kafka.zk.KafkaZkClient$CheckedEphemeral.create(KafkaZkClient.scala:1722)
at kafka.zk.KafkaZkClient.checkedEphemeralCreate(KafkaZkClient.scala:1689)
at kafka.zk.KafkaZkClient.registerBroker(KafkaZkClient.scala:97)
at kafka.server.KafkaServer.startup(KafkaServer.scala:260)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
at kafka.Kafka$.main(Kafka.scala:75)
at kafka.Kafka.main(Kafka.scala)
解决方案
在 debezium Kafka 容器中,您需要通过-e BROKER_ID=2
. 默认值为 1。
https://github.com/debezium/docker-images/blob/master/kafka/0.10/docker-entrypoint.sh#L6-L9
在其他 Kafka 容器中,KAFKA_BROKER_ID
您可以使用一个变量,每个容器需要不同的变量,该变量还将内部主题的复制因子设置为 2,并正确设置广告侦听器。
这是一个示例撰写文件
---
version: '2'
services:
zookeeper:
image: confluentinc/cp-zookeeper:latest
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
kafka-1:
image: confluentinc/cp-kafka:latest
depends_on:
- zookeeper
ports:
- 19092:19092
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT_HOST://localhost:19092,PLAINTEXT://kafka-1:9092
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 2
kafka-2:
image: confluentinc/cp-kafka:latest
depends_on:
- zookeeper
ports:
- 29092:29092
environment:
KAFKA_BROKER_ID: 2
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT_HOST://localhost:29092,PLAINTEXT://kafka-2:9092
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 2
前任。
$ kafkacat -L -b localhost:19092
Metadata for all topics (from broker 1: localhost:19092/1):
2 brokers:
broker 2 at localhost:29092
broker 1 at localhost:19092 (controller)
1 topics:
topic "__confluent.support.metrics" with 1 partitions:
partition 0, leader 1, replicas: 1,2, isrs: 1,2
注意:在一台机器上运行多个代理不会提高 Kafka 的任何性能或可靠性
推荐阅读
- google-cloud-platform - Google Cloud Spanner 是否支持索引交集/组合/合并?
- npm - `npm run watch` 不在开发模式下编译,但在生产中编译(-p 标志)
- python-3.x - 访问列表中的字典元素
- python - Python如何阻止atom-beautify包进行分行?
- python-3.x - 如何标准化元组中的分布?
- android - 以编程方式打开另一个应用程序然后关闭它会导致主应用程序冻结
- mysql - 显示 id 并计算相同的外键
- reactjs - 我的 React 网站中的 React 可加载问题
- unit-testing - 单元测试在 xunit.runner.console 中失败,但在本地通过
- odoo - “group_by_no_leaf”有什么用?