elasticsearch - Logstash JSON Grok 过滤器问题
问题描述
我设置了 squid 代理,通过 Logstash 将 JSON 格式的日志发送到 Elastic。我正在尝试使用 GROK 过滤来解析日志。该过滤器在 Kiabana Grok 调试器中工作,但在我重新启动 Logstash 时抱怨以下错误
Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:squid_logs,
:exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"
{\", \",\", \"]\" at line 10, column 62 (byte 137) after filter {\n grok {\n match => {\n
\"message\" => [ \"%{IPV4:vendor_ip}\", \"%{WORD:message}\"", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'",
"org/logstash/execution/AbstractPipelineExt.java:184:in `initialize'",
"org/logstash/execution/JavaBasePipelineExt.java:69:in `initialize'",
"/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in `initialize'",
"/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'",
"/usr/share/logstash/logstash-core/lib/logstash/agent.rb:389:in `block in converge_state'"]}
我有以下 GROK 过滤器
"%{IPV4:vendor_ip}", "%{WORD:message}": "%{IPV4:clientip}", "%{WORD:message}": "%
{DATA:timestamp}", "%{WORD:message}": "%{WORD:verb}", "%{WORD:message}": "%{DATA:request}", "%
{WORD:message}": "%{URIPATHPARAM:path}"
在 Kibana Grok 调试器中,过滤器可以很好地处理如下消息:
{ "vendor_ip": "x.x.x.x", "clientip": "x.x.x.x", "timestamp": "2021-04-09T13:58:38+0000",
"verb": "GET", "request": "https://domain", "path": "/somepath", "httpversion": "HTTP/1.1",
"response": 200, "bytes": 2518042, "referer": "-", "useragent": "Microsoft BITS/7.8",
"request_status": "HIER_DIRECT", "hierarchy_status": "HIER_DIRECT" }
Logstash 配置如下:
input {
beats {
port => 5045
}
}
filter {
grok {
match => {
"message" => [ "%{IPV4:vendor_ip}", "%{WORD:message}": "%{IPV4:clientip}", "%{WORD:message}": "%{DATA:timestamp}", "%{WORD:message}": "%{WORD:verb}", "%{WORD:message}": "%{DATA:request}", "%{WORD:message}": "%{URIPATHPARAM:path}" ]
}
}
}
output {
elasticsearch {
hosts => ["x.x.x.x:9200"]
index => "squid_logs"
}
}
解决方案
使用 grok 过滤器解析 json 消息是错误的方法,没有必要这样做,而且工作量很大,因为您需要转义消息中的所有双引号,否则会出现配置错误,即你的情况。
使用json过滤器解析 json 消息
只需在您的管道中使用它:
filter {
json {
source => "message"
}
}
推荐阅读
- javascript - 如何使用javascript比较css中的当前宽度?
- go - 为什么指针赋值会导致变量赋值不总是坚持?
- reactjs - 如何为每一行添加单独的滚动,我想在 ag-grid-react 的嵌套级别排序
- javascript - How to perform multiple axios POST requests in a For Loop with different body data for each iteration of loop?
- python - 如何从列表中匹配按字典顺序提到的键
- javascript - 分离两个 ID
- javascript - Javascript不会在html中将数据添加到表中
- django - Django:如何使用 ListView 将最后添加的日期添加到查询集中
- javascript - 使用 ajax 和 jquery 在烧瓶中构建类似的帖子失败
- typescript - 替代 Datasnapshot 的 foreach 循环,以便在满足条件时退出循环?