logging - Fluentd 无法与 Kafka 代理建立连接:Errno::ECONNRESET: Connection reset by peer
问题描述
我已经配置了一个fluentd
边车容器来将日志发送到kafka
.
这是我的fluent.conf
文件:
<source>
@type forward
bind 127.0.0.1
port 24224
</source>
<match log.default>
@type kafka2
# list of seed brokers
brokers node1:9092, node2:9092, node3:9092
use_event_time true
# buffer settings
buffer_type file
buffer_path /var/log/td-agent/buffer/kafka
flush_interval 3s
# topic settings
default_topic app_logs
# data type settings
<format>
@type json
</format>
# producer settings
max_send_retries 1
required_acks -1
#deubg log
@log_level debug
</match>
fluentd
可以从运行 sidecar 容器的 pod 访问代理主机。但是,sidecar 容器无法连接到代理。它返回以下日志:
2020-10-23 11:41:51 +0000 [info]: starting fluentd-1.10.4 pid=8 ruby="2.5.8"
2020-10-23 11:41:51 +0000 [info]: spawn command to main: cmdline=["/usr/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "--under-supervisor"]
2020-10-23 11:41:55 +0000 [info]: adding match pattern="log.default" type="kafka2"
2020-10-23 11:41:56 +0000 [info]: #0 brokers has been set: ["node1:9092", "node2:9092", "node3:9092"]
2020-10-23 11:41:56 +0000 [info]: adding source type="forward"
2020-10-23 11:41:56 +0000 [info]: #0 starting fluentd worker pid=22 ppid=8 worker=0
2020-10-23 11:41:56 +0000 [debug]: #0 buffer started instance=47608408649900 stage_size=0 queue_size=0
2020-10-23 11:41:56 +0000 [info]: #0 initialized kafka producer: fluentd
2020-10-23 11:41:56 +0000 [info]: #0 listening port port=24224 bind="127.0.0.1"
2020-10-23 11:41:56 +0000 [info]: #0 fluentd worker is now running worker=0
2020-10-23 11:41:57 +0000 [debug]: #0 flush_thread actually running
2020-10-23 11:41:57 +0000 [debug]: #0 enqueue_thread actually running
2020-10-23 11:43:02 +0000 [warn]: #0 Send exception occurred: Could not connect to any of the seed brokers:
- kafka://node1:9092: Connection error Errno::ECONNRESET: Connection reset by peer
- kafka://node2:9092: Connection error Errno::ECONNRESET: Connection reset by peer
- kafka://node3:9092: Connection error Errno::ECONNRESET: Connection reset by peer
2020-10-23 11:43:02 +0000 [warn]: #0 Exception Backtrace : /usr/lib/ruby/gems/2.5.0/gems/ruby-kafka-1.3.0/lib/kafka/cluster.rb:448:in `fetch_cluster_info'
如何摆脱这个问题?这可能是代理节点方面的问题吗?
解决方案
在 fluentd 主机上,尝试使用https://github.com/edenhill/kafkacat调试 Kafka 连接/访问。我确定这是网络/防火墙问题,而不是流利的。
推荐阅读
- eclipse - 如何在具有不同拱门的远程机器上分析特定线程?
- javascript - 变量被分配了一个值但从未使用过,除非它实际被使用
- html - React - 无法让 CDN 字体在 Safari 浏览器上工作
- c# - 将 RichTextBox 绑定到 FlowDocumentReader 内部的内容时,它变为 ReadOnly
- javascript - 以高分辨率png保存html div
- spring - spring security,从表中获取方法和 url 并动态验证 authorizeRequests
- r - 从同级多个同名节点抓取网页
- android - React Native Azure Auth 注销而不重定向到网页
- sql - 如何根据特定字段更新具有数字总和的表?
- javascript - 使用 jest-puppeteer 对文本字段进行 Webscrape 测试