时间戳

首页 > 解决方案 > 如何在logstash中拆分多行日志

logstash - 如何在logstash中拆分多行日志

问题描述

我正在尝试为我的日志文件创建过滤器。我的日志是：

    =-=-=-=-=-=-=-=-
    Timestamp: Thursday, April 19, 2018 2:48:49 AM
    Message: HandlingID: 3
    An exception of type 'System.Exception' occurred and was caught.
    ----------------------------------------------------------------
    04/19/2018 02:48:49
    Type : System.Exception,ib, Version=4.0.0.0, Culture=neutral
    Message : TRY
    Source : 
    Help link : 
    Data : LinkedList
    TargetSite : 
    HResult : LALA
    Stack Trace : The stack trace is unavailable.
    Additional Info:

    MachineName : S
    TimeStamp : 4/19/2018 6:48:49 AM
    FullName : Some Info
    AppDomainName : AA
    ThreadIdentity : 
    WindowsIdentity : jj

    Category: Error
    Priority: 0
    EventId: 1
    Severity: Extreme
    Title:p
    Machine: S
    Application Domain: y
    Process Id: 
    Process Name: l
    Win32 Thread Id: 6
    Thread Name: 
    Extended Properties: 
    =-=-=-=-=-=-=-=-
    =-=-=-=-=-=-=-=-

//SIMILAR LOG
    =-=-=-=-=-=-=-=-

这里 =-=-=-=-=-=-=-=- 表示新日志的结束和开始。我的配置文件是：

input {
beats {
        port => "5044"
    }
}
filter {
multiline {
       pattern => "^=-=-=-=-=-=-=-=-"
       negate => true
       what => previous
    }
}
output {
elasticsearch {
        hosts => [ "localhost:9200" ]
    }
}

这是为每个日志创建 2 个文档（行）。一个包含分隔符+日志，另一个仅包含分隔符。我想删除仅包含分隔符的文档。另外，请让我知道如何将我的日志拆分为不同的字段，以及是否有任何适当的文档可以帮助我形成过滤器。我是logstash的新手。

标签： logstashlogstash-grok

解决方案

如果要包含所有新行并在每次出现 时拆分日志=-=-=-=-=-=-=-=-，可以将其与多行修饰符匹配，(?m)如下所示，

(?m)%{GREEDYDATA:log}=-=-=-=-=-=-=-=-

它会输出，

{
  "log": [
    [
      "    Timestamp: Thursday, April 19, 2018 2:48:49 AM\n    Message: HandlingID: 3\n    An exception of type 'System.Exception' occurred and was caught.\n    ----------------------------------------------------------------\n    04/19/2018 02:48:49\n    Type : System.Exception,ib, Version=4.0.0.0, Culture=neutral\n    Message : TRY\n    Source : \n    Help link : \n    Data : LinkedList\n    TargetSite : \n    HResult : LALA\n    Stack Trace : The stack trace is unavailable.\n    Additional Info:\n\n    MachineName : S\n    TimeStamp : 4/19/2018 6:48:49 AM\n    FullName : Some Info\n    AppDomainName : AA\n    ThreadIdentity : \n    WindowsIdentity : jj\n\n    Category: Error\n    Priority: 0\n    EventId: 1\n    Severity: Extreme\n    Title:p\n    Machine: S\n    Application Domain: y\n    Process Id: \n    Process Name: l\n    Win32 Thread Id: 6\n    Thread Name: \n    Extended Properties: \n    "
    ]
  ]
}

您可以在https://grokdebug.herokuapp.com/对其进行测试

推荐阅读