docker - 在 docker 中运行 clickhouse 和 Kafka
问题描述
我正在尝试将 Kafka 与 clickhouse 集成。我想将一些数据从我的 REST API 发送到 kafka,并将其传递给 clickhouse,这将是我的数据库。到目前为止,我能够让 Kafka 单独工作并将数据从一端传递到另一端。但是当我尝试在 docker 中使用它时,我遇到了一些问题。好吧,我可以使用 kafka 工具看到在 kafka 端消耗的数据。但我的 clickhouse 显然没有连接。这是错误,我在运行 docker 时不断收到错误消息:
clickhouse-server_1 | %3|1584611439.583|FAIL|ClickHouse 20.3.3.6#consumer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 1ms in state CONNECT)
clickhouse-server_1 | %3|1584611439.583|ERROR|ClickHouse 20.3.3.6#consumer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 1ms in state CONNECT)
clickhouse-server_1 | %3|1584611439.583|ERROR|ClickHouse 20.3.3.6#consumer-1| [thrd:localhost:9092/bootstrap]: 1/1 brokers are down
clickhouse-server_1 | %3|1584611439.583|FAIL|ClickHouse 20.3.3.6#consumer-2| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
clickhouse-server_1 | %3|1584611439.583|ERROR|ClickHouse 20.3.3.6#consumer-2| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
clickhouse-server_1 | %3|1584611439.583|ERROR|ClickHouse 20.3.3.6#consumer-2| [thrd:localhost:9092/bootstrap]: 1/1 brokers are down
我还在下面添加了日志文件:
2020.03.19 10:16:11.637331 [ 1 ] {} <Error> Application: Listen [::]:8123 failed: Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = DNS error: EAI: -9 (version 20.3.3.6 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2020.03.19 10:16:11.637658 [ 1 ] {} <Error> Application: Listen [::]:9000 failed: Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = DNS error: EAI: -9 (version 20.3.3.6 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2020.03.19 10:16:11.637854 [ 1 ] {} <Error> Application: Listen [::]:9009 failed: Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = DNS error: EAI: -9 (version 20.3.3.6 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2020.03.19 10:16:11.678748 [ 74 ] {} <Error> void DB::DDLWorker::runMainThread(): Code: 999, e.displayText() = Coordination::Exception: All connection tries failed while connecting to ZooKeeper. Addresses: 192.168.144.3:2181
Poco::Exception. Code: 1000, e.code() = 111, e.displayText() = Connection refused (version 20.3.3.6 (official build)), 192.168.144.3:2181
Poco::Exception. Code: 1000, e.code() = 111, e.displayText() = Connection refused (version 20.3.3.6 (official build)), 192.168.144.3:2181
Poco::Exception. Code: 1000, e.code() = 111, e.displayText() = Connection refused (version 20.3.3.6 (official build)), 192.168.144.3:2181
(Connection loss), Stack trace (when copying this message, always include the lines below):
0. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x102d352c in /usr/bin/clickhouse
1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x8f2d989 in /usr/bin/clickhouse
2. Coordination::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, int) @ 0xdd300e4 in /usr/bin/clickhouse
3. Coordination::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0xdd3070e in /usr/bin/clickhouse
4. ? @ 0xdd638b2 in /usr/bin/clickhouse
5. Coordination::ZooKeeper::ZooKeeper(std::__1::vector<Poco::Net::SocketAddress, std::__1::allocator<Poco::Net::SocketAddress> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Poco::Timespan, Poco::Timespan, Poco::Timespan) @ 0xdd6099b in /usr/bin/clickhouse
6. zkutil::ZooKeeper::init(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0xdd3fd53 in /usr/bin/clickhouse
7. zkutil::ZooKeeper::ZooKeeper(Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0xdd405bd in /usr/bin/clickhouse
8. DB::Context::getZooKeeper() const @ 0xce5fb97 in /usr/bin/clickhouse
9. DB::DDLWorker::getAndSetZooKeeper() @ 0xce84c53 in /usr/bin/clickhouse
10. DB::DDLWorker::runMainThread() @ 0xce8eae3 in /usr/bin/clickhouse
11. ThreadFromGlobalPool::ThreadFromGlobalPool<void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&)::'lambda'()::operator()() const @ 0xce96031 in /usr/bin/clickhouse
12. ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0x8f50b07 in /usr/bin/clickhouse
13. ? @ 0x8f4f00f in /usr/bin/clickhouse
14. start_thread @ 0x76db in /lib/x86_64-linux-gnu/libpthread-2.27.so
15. __clone @ 0x12188f in /lib/x86_64-linux-gnu/libc-2.27.so
(version 20.3.3.6 (official build))
以下是我正在使用的一些配置:
CREATE TABLE IF NOT EXISTS filters (
name String,
value String
) ENGINE = Kafka SETTINGS
kafka_broker_list = 'kafka://localhost:9092',
kafka_topic_list = 'testtopic',
kafka_group_name = 'test',
kafka_format = 'JSONEachRow',
kafka_num_consumers = 2
我的卡夫卡引导部分:
producer = KafkaProducer(bootstrap_servers=['kafka://localhost:9092'], value_serializer=lambda m: json.dumps(m).encode('ascii'))
还有我的 docker 配置部分:
version: "3.6"
services:
clickhouse-server:
image: yandex/clickhouse-server
volumes:
- ./clickhouse/config.xml:/etc/clickhouse-server/config.xml
- ./clickhouse/zookeeper-servers.xml:/etc/clickhouse-server/conf.d/zookeeper-servers.xml
- ./shared/ch-data/clickhouse:/var/lib/clickhouse
depends_on:
- kafka
links:
- kafka
ports:
- 9000:9000
clickhouse-client:
image: yandex/clickhouse-client
entrypoint:
- /bin/sleep
command:
- infinity
kafka:
image: wurstmeister/kafka:2.11-1.0.2
volumes:
- ./shared/ch-data/kafka:/data
environment:
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: 192.168.1.86
links:
- zookeeper
ports:
- 9092:9092
- 9094:9094
zookeeper:
image: zookeeper
volumes:
- ./shared/ch-data/zookeeper:/data
ports:
- 2181:2181
我对 bootstrap_Servers 和 KAFKA_ADVERTISED_LISTENERS 部分感到特别困惑。据我所知,我已将我的主机 IP 提供为 KAFKA_ADVERTISED_LISTENER,据我了解这里
解决方案
似乎您误解了 Docker 网络。最重要的是,links
功能已被弃用,所以我不确定你从哪里复制了你的撰写文件
localhost
将始终是当前容器,而不是另一个
修复:kafka://localhost:9092
&bootstrap_servers=['kafka://localhost:9092']
使用实际的 Docker 服务名称
据我所知,我已将我的主机 IP 提供为 KAFKA_ADVERTISED_LISTENER
- 如果您使用多台物理机,则只需要主机 IP。
- 如果您在同一台物理机器上使用 Docker 之外的 Kafka 客户端,则只需宣传 localhost
- 您应该始终为 Docker 网络中的客户端公布 kafka 容器服务名称
总而言之,KAFKA_LISTENERS
将有 3-4 个端口,第四个是用于复制的跨安全协议。你做多少广告取决于你
并且您将添加协议映射,如该博客文章中所述。
请注意,您不需要使用任何特定的 kafka 或 Zookeeper docker 映像,并且博客中提供的那些 compose 文件可以正常工作。另外,永远不要相信两个不同的 Docker 镜像共享同一组环境变量。
wurstmeister 图片有自己的 Github 页面,讨论听众
推荐阅读
- reactjs - 我无法立即获得更新的 internalStorage 值来更新我在 Reactjs 中的购物车通知
- python - 如何根据 Python Pandas 列表中的值创建新列?
- amazon-web-services - 预签名 s3 文件时没有这样的存储桶错误
- node.js - Nodemailer 在 less1 秒内发送了两次电子邮件
- pine-script - 当 30 分钟蜡烛收盘时,每 50 美元的高点和低点创建 line.new
- java - 是否可以根据步骤定义返回的数组列表迭代一组特征文件行
- java - 如何在 RTL 模式下的视图寻呼机中实现立方体效果?
- javascript - 编辑csv文件nodejs电子中的一行
- python - 如何通过递归查找包含嵌套字典中的键的字典?
- python - 将字符串列表与列匹配并返回对应的列值