时间戳

首页 > 解决方案 > Logstash grok 过滤器 - 字段值重复

logstash - Logstash grok 过滤器 - 字段值重复

问题描述

我的logstash过滤器配置如下：

filter {
  grok {
    patterns_dir => ["/usr/share/logstash/pipeline/patterns/"]
    match => {
      "[message]" => "%{TIMESTAMP_ISO8601:timestamp} %{THREAD:thread} %{LOGLEVEL:level} %{LOGGER:logger} %{CONTEXT:context} - %{GREEDYDATA:message}"
    }
  }
  mutate {
    rename => { "[fields][index]" => "application" }
    rename => { "[host][name]" => "instance" }
    remove_field => ["@version","agent.ephemeral_id","agent","ecs","fields","input","tags"]
  }
}

Grok 调试器提示一切正常，对于错误行：

2020-10-28 05:14:41,282 [Worker-5] DEBUG Amount  - calculate operation: [1], useCurrencyCodeOfPosition: [false]

我得到以下输出：

{
  "level": "DEBUG",
  "logger": "Amount",
  "context": "",
  "thread": "Worker-5",
  "message": "calculate operation: [1], useCurrencyCodeOfPosition: [false]",
  "timestamp": "2020-10-28 05:14:41,282"
}

模式定义如下：

THREAD \[(?<thread>[^\]]*)\]
LOGGER (?<logger>[^ ]*)
CONTEXT (?<context>[^-]*)

现在，由 grok 过滤器生成的每个值都被复制，如下例所示：

             "logger" => [
    [0] "Amount",
    [1] "Amount"
],
             "thread" => [
    [0] "[Worker-5]",
    [1] "Worker-5"

这里有什么问题？我就是想不通。这是我的第一个过滤器:)。我正在使用 Logstash 7.9.2 (dockerized)

标签： logstashlogstash-grok

解决方案

我认为过滤器中的自定义模式存在问题。你想要的也可以简单地使用下面的开箱即用模式来实现

filter{
 grok {
    match => {
      "message" => "%{TIMESTAMP_ISO8601:timestamp}%{SPACE}\[%{DATA:thread}\]%{SPACE}%{LOGLEVEL:level}%{SPACE}%{NOTSPACE:logger} %{DATA:context}-%{SPACE}%{GREEDYDATA:message}"
    }
    overwrite => [ "message" ]
  }
  mutate {
    rename => { "[fields][index]" => "application" }
    rename => { "[host][name]" => "instance" }
    remove_field => ["@version","agent.ephemeral_id","agent","ecs","fields","input","tags"]
  }
}

查看默认grok模式的此链接。如果您需要对这些事件进行时间序列分析，我建议您@timestamp使用timestamp或至少应用日期过滤器覆盖timestamp.

如果您希望捕获多行堆栈跟踪错误，请考虑在输入插件上使用多行过滤器。

推荐阅读