centos - 连接回退失败(async(tcp://ip:5044)): dial tcp ip:5044: i/o timeout
问题描述
Filebeat 在机器 B 上运行,它读取日志并推送到机器 A 上的 ELK logstash。
但是在机器B的filebeat日志中,它显示了错误i/o timeout
2019-08-24T12:13:10.065+0800 ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://example.com:5044)): dial tcp xx.xx.xx.xx:5044: i/o timeout
2019-08-24T12:13:10.065+0800 INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://example.com:5044)) with 1 reconnect attempt(s)
我检查了机器 A 上运行良好的 logstash,可以监听0 0.0.0.0:5044
这是logstash日志
[INFO ] 2019-08-24 12:09:35.217 [[main]-pipeline-manager] beats - Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
这是netstat
输出,
$ sudo netstat -tlnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:5044 0.0.0.0:* LISTEN 20668/java
我还检查了机器 A 上的防火墙是否已关闭。
$ firewall-cmd --list-all
FirewallD is not running
$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy DROP)
target prot opt source destination
我也使用 telnet 连接机器 A,但我明白了,
$ telnet example.com 5044
Trying xx.xx.xx.xx...
telnet: connect to address xx.xx.xx.xx: Connection timed out
我在机器 A(本地)上运行具有相同配置的 filebeat 以检查机器 B(远程)上的 filebeat 配置是否错误,它运行良好。
2019-08-24T14:17:35.195+0800 INFO pipeline/output.go:95 Connecting to backoff(async(tcp://localhost:5044))
2019-08-24T14:17:35.198+0800 INFO pipeline/output.go:105 Connection to backoff(async(tcp://localhost:5044)) established
解决方案
最后发现是VPS Provider aliyun造成的,它只开放了22、80443等一些常用端口。
我需要登录阿里云VPS管理页面,打开5044让VPS Provider绕过5044端口。
*注:*附件:我在使用 ELK 配置 filebeat 时遇到的一些其他问题。
**问题 1:** 连接回退失败(async(tcp://ip:5044)): dial tcp ip:5044: connect: connection denied
2019-08-26T10:25:41.955+0800 ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://example.com:5044)): dial tcp xx.xx.xx.xx:5044: connect: connection refused
2019-08-26T10:25:41.955+0800 INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://example:5044)) with 2 reconnect attempt(s)
问题2:无法发布事件导致:write tcp ip:46890->ip:5044: write: connection reset by peer
2019-08-26T10:28:32.274+0800 ERROR logstash/async.go:256 Failed to publish events caused by: write tcp xx.xx.xx.xx:46890->xx.xx.xx.xx:5044: write: connection reset by peer
2019-08-26T10:28:33.311+0800 ERROR pipeline/output.go:121 Failed to publish events: write tcp xx.xx.xx.xx:46890->xx.xx.xx.xx:5044: write: connection reset by peer
问题 3: Filebeat 错误:lumberjack 协议错误和 Logstasherror: OPENSSL_internal:WRONG_VERSION_NUMBER
Filebeat 日志错误,
2019-08-26T08:49:09.505+0800 INFO pipeline/output.go:95 Connecting to backoff(async(tcp://example.com:5044))
2019-08-26T08:49:09.588+0800 INFO pipeline/output.go:105 Connection to backoff(async(tcp://example.com:5044)) established
2019-08-26T08:49:09.605+0800 ERROR logstash/async.go:256 Failed to publish events caused by: lumberjack protocol error
2019-08-26T08:49:09.606+0800 ERROR logstash/async.go:256 Failed to publish events caused by: client is not connected
Logstash 日志,
[INFO ] 2019-08-26 08:49:29.444 [defaultEventExecutorGroup-4-2] BeatsHandler - [local: 0.0.0.0:5044, remote: undefined] Handling exception: javax.net.ssl.SSLHandshakeException: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER
[WARN ] 2019-08-26 08:49:29.445 [nioEventLoopGroup-2-7] DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
...
这三个问题都是由于未配置造成的,这里是可行的配置,
logstash 版本,
/usr/share/logstash/bin/logstash -V
logstash 7.3.1
文件节拍版本,
/usr/share/filebeat/bin/filebeat version
filebeat version 7.3.1 (amd64), libbeat 7.3.1 [a4be71b90ce3e3b8213b616adfcd9e455513da45 built 2019-08-19 19:30:50 +0000 UTC]
logstash 配置文件/etc/logstash/conf.d/beat.conf
input {
beats {
port => 5044
ssl => true
ssl_certificate_authorities => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
ssl_verify_mode => "peer"
}
}
output {
elasticsearch {
hosts => "http://127.0.0.1:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
文件节拍配置文件/etc/filebeat/filebeat.yml
#=========================== Filebeat inputs =============================
filebeat.inputs:
# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.
- type: log
# Change to true to enable this input configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /data/error_logs/Log_error_201908
#----------------------------- Logstash output --------------------------------
output.logstash:
# The Logstash hosts
hosts: ["example.com:5044"]
# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
ssl.certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]
# Certificate for SSL client authentication
ssl.certificate: "/etc/pki/tls/certs/logstash-forwarder.crt"
# Client Certificate Key
ssl.key: "/etc/pki/tls/private/logstash-forwarder.key"
推荐阅读
- typo3 - TYPO3 从 v9 升级到 V10 延迟加载错误
- r - R - Timeseries、autoplot()、scale_y_continuous() 和消失的 geom_abline()
- three.js - 移动设备相机时如何将对象粘贴在three.js中的某个位置
- javascript - 将 javascript 函数的结果返回到 Blazor 服务器中的变量
- python - 为什么我在 KFold 中得到不同数量的火车尺寸
- git - 使用命令行保护主分支并合并拉取请求
- algorithm - 快速查找重叠范围的数据结构?(例如日期、数字)
- typescript - 如何使用 TypeORM 实现三重关系
- python - Webscraping 运动鞋的尺码
- caching - 是否可以远程列出 Redis 缓存中的所有活动键?