elasticsearch - Logstash un-gzip 阵列日志配置
问题描述
每个人!我有 logstash 配置,它将日志从 RabbitMQ 转发到 elasticSearch。像这样的东西:
input {
rabbitmq {
...
}
}
filter {
if [type] == "rabbitmq" {
json {
source => "message"
target => "message"
}
}
}
output {
elasticsearch {
hosts => ["${ES_HOST}"]
user => "${ES_USERNAME}"
password => "${ES_PASSWORD}"
sniffing => false
index => "kit_events-%{[message][elasticsearch][index]}"
}
}
我们被迫即时压缩日志,因为它们花费了太多流量。日志被移入数组并压缩。配置解压缩并将数组拆分回对象的正确方法是什么?
我做了一些研究,发现 Ruby(?) 上有 gzip_lines 插件和一些东西来解析数组,但我没有实现它。以前有人做过这样的东西吗?
升级版:
添加了此过滤器
filter {
if [type] == "kitlog-rabbitmq" {
ruby {
init => "
require 'zlib'
require 'stringio'
"
code => "
body = event.get('[http][response][body]').to_s
sio = StringIO.new(body)
gz = Zlib::GzipReader.new(sio)
result = gz.read.to_s
event.set('[http][response][body]', result)
"
}
}
}
现在发现错误
ERROR][logstash.filters.ruby ] Ruby exception occurred: not in gzip format
[DEBUG][logstash.pipeline ] output received {"event"=>{"@timestamp"=>2018-11-30T09:16:19.127Z, "tags"=>["_rubyexception"], "@version"=>"1", "message"=>"x^\\x8B\\xAEV*\\xCE\\xCE\\xCC\\xC9)V\\xB2R\\x88V\\xD26T07\\xB7\\xB0\\xB4\\xB44000W\\x8A\\xD5QPJ\\xCE\\xCF+IL.\\u0001\\xCA*)\\u0001\\xB9\\xA9\\xB9\\x89\\x999 N\\x96C\\x96^r~.X,\\xA5\\u0014(R\\xADT\\x9A\\u000E6#\\xA0\\xB2$#?\\u000F\\xAC\\xB9\\u0000\\\"\\xE2\\u001C\\xAC\\u0014[\\v\\xE4\\xE6%概\\xF4z\\u0001\\xE9b%\\xA0\\xC8\\xC0\\xD9\\u001D\\v\\u0000\\u0003\\x9ADk", "type"=>"kitlog-rabbitmq"}}
正在尝试不同的 gzipping 方法,但结果仍然相同。还尝试更改输入编解码器(纯 - utf-8,纯 - 二进制)