elasticsearch - 无法使用 logstash (7.6.2) 和 xml 过滤器插件设置解析和设置时间戳
问题描述
环境详情：ELK stack 7.6.2 Windows 10
通过 Logstash 进行索引时，我无法从我的日志中替换/设置弹性搜索时间戳。它只是作为一个新字段添加，而不是替换原始字段。它只是添加了“_dateparsefailure”标签而没有任何其他信息。
我怀疑日期过滤器不起作用。
我的示例日志数据：
<log4j:event logger="SomeOrganization.Shared.ApplicationBlocks.Logging.Logger" timestamp="1530819710045" level="WARN" thread="27"><log4j:message>registrarCheck.bookingWizardController.TryUpdatebookingCareOptions(): bookingCareOptionId: CenterBasedCare, bookingId: 5745493, bookingregistrarsCount: 5, IsEditbooking: False, IsEditbookingStep2Modified: False, IsMemberShip: False</log4j:message><log4j:properties><log4j:data name="log4japp" value="/LM/W3SVC/2/ROOT-1-131752914805620482" /><log4j:data name="log4net:Identity" value="user1" /><log4j:data name="log4jmachinename" value="webserver1" /><log4j:data name="log4net:UserName" value="SomeOrganization\!svc-app-identity" /><log4j:data name="log4net:HostName" value="webserver1" /></log4j:properties><log4j:locationInfo class="SomeOrganization.Shared.ApplicationBlocks.Logging.Logger" method="Warn" file="c:\Builds\5\mainline\Main.SomeApplication\Sources\mainline\Main\Shared\SomeOrganization.Shared.ApplicationBlocks\Logging\Logging.cs" line="283" /></log4j:event>
<log4j:event logger="SomeOrganization.Shared.ApplicationBlocks.Logging.Logger" timestamp="1530819760731" level="ERROR" thread="15"><log4j:message>ERROR from EasyDraft API for funding accountid->0->Name: firstname lastname->Card number is invalid</log4j:message><log4j:properties><log4j:data name="log4japp" value="/LM/W3SVC/2/ROOT-1-131752914805620482" /><log4j:data name="log4net:Identity" value="user1" /><log4j:data name="log4jmachinename" value="webserver1" /><log4j:data name="log4net:UserName" value="SomeOrganization\!svc-app-identity" /><log4j:data name="log4net:HostName" value="webserver1" /></log4j:properties><log4j:locationInfo class="SomeOrganization.Shared.ApplicationBlocks.Logging.Logger" method="Error" file="c:\Builds\5\mainline\Main.SomeApplication\Sources\mainline\Main\Shared\SomeOrganization.Shared.ApplicationBlocks\Logging\Logging.cs" line="139" /></log4j:event>
<log4j:event logger="SomeOrganization.Shared.ApplicationBlocks.Logging.Logger" timestamp="1530819760856" level="ERROR" thread="15"><log4j:message>Error in controller: effective username: user1, identity username: user1, machine name: webserver1
Client Name: [zzz Test ESomeApplication], Contract Id: [7ee17d62-d292-e511-b173-005056991898]
, Person Id: [143658262]
, Client ID: [b33442b3-d192-e511-b173-005056991898], Contract Relationship ID: [4529625]
, Person Fullname: [firstname lastname].
, Full Name: [firstname lastname], CRM ID: [a64c97b1-8a80-e811-b738-005056991899]</log4j:message><log4j:properties><log4j:data name="log4japp" value="/LM/W3SVC/2/ROOT-1-131752914805620482" /><log4j:data name="log4net:Identity" value="user1" /><log4j:data name="log4jmachinename" value="webserver1" /><log4j:data name="log4net:UserName" value="SomeOrganization\!svc-app-identity" /><log4j:data name="log4net:HostName" value="webserver1" /></log4j:properties><log4j:throwable>SomeOrganization.SomeApplication.BusinessLogic.Security.SomeOrganizationSomeApplicationException: Exception of type 'SomeOrganization.SomeApplication.BusinessLogic.Security.SomeOrganizationSomeApplicationException' was thrown.
at SomeOrganization.SomeApplication.BusinessLogic.PaymentAccount.Save() in c:\Builds\5\mainline\Main.SomeApplication\Sources\mainline\Main\SomeApplication\SomeOrganization.SomeApplication.BusinessLogic\PaymentAccount.cs:line 415
at Csla.BusinessBase`1.Save(Boolean forceUpdate) in C:\andre\mainline\SomeApplicationNewDevelopment\CSLA\Source-4.3.12\Csla\BusinessBase.cs:line 163
at Csla.BusinessBase`1.Csla.Core.ISavable.Save(Boolean forceUpdate) in C:\andre\mainline\SomeApplicationNewDevelopment\CSLA\Source-4.3.12\Csla\BusinessBase.cs:line 350
at SomeOrganization.Shared.Web.ApplicationBlocks.Controllers.CustomCslaMvcController.SaveObject[T](T item, Action`1 updateModel, Boolean forceUpdate) in c:\Builds\5\mainline\Main.SomeApplication\Sources\mainline\Main\Shared\SomeOrganization.Shared.Web.ApplicationBlocks\Controllers\CustomCslaMvcController.cs:line 171</log4j:throwable><log4j:locationInfo class="SomeOrganization.Shared.ApplicationBlocks.Logging.Logger" method="Error" file="c:\Builds\5\mainline\Main.SomeApplication\Sources\mainline\Main\Shared\SomeOrganization.Shared.ApplicationBlocks\Logging\Logging.cs" line="165" /></log4j:event>
<log4j:event logger="SomeOrganization.Shared.ApplicationBlocks.Logging.Logger" timestamp="1530824089499" level="ERROR" thread="41"><log4j:message>Error Occured while Save Login in Class Login & Method : Save For Username : tegh14</log4j:message><log4j:properties><log4j:data name="log4japp" value="/LM/W3SVC/2/ROOT-2-131752976869399121" /><log4j:data name="log4net:UserName" value="SomeOrganization\!svc-app-identity" /><log4j:data name="log4jmachinename" value="webserver1" /><log4j:data name="log4net:HostName" value="webserver1" /></log4j:properties><log4j:throwable>System.Security.Authentication.AuthenticationException: We can�t find that username and/or password. If you are trying to register for the first time using your employer�s credentials, select the Create Your Profile link below. If you are having trouble accessing the site, feel free to call us at none-one-CARES in the United States or Canada, 0800 000 000 in the United Kingdom, or 0800 000 000 in Ireland.
at SomeOrganization.Shared.BusinessLogic.Security.Login.Save() in c:\Builds\5\mainline\Main.SomeApplication\Sources\mainline\Main\Shared\SomeOrganization.Shared.BusinessLogic\Security\Login.cs:line 547</log4j:throwable><log4j:locationInfo class="SomeOrganization.Shared.ApplicationBlocks.Logging.Logger" method="Error" file="c:\Builds\5\mainline\Main.SomeApplication\Sources\mainline\Main\Shared\SomeOrganization.Shared.ApplicationBlocks\Logging\Logging.cs" line="165" /></log4j:event>
<log4j:event logger="SomeOrganisation.Shared.ApplicationBlocks.Logging.Logger" timestamp="1587880949425" level="WARN" thread="47"><log4j:message>User mphilpunla->LoginWithSAML->lobuniqueId 19153694</log4j:message><log4j:properties><log4j:data name="log4jmachinename" value="webserver2" /><log4j:data name="log4japp" value="/LM/W3SVC/2/ROOT-1-132323544167926323" /><log4j:data name="log4net:UserName" value="SomeOrganisation\!svc-lob-apps" /><log4j:data name="log4net:Identity" value="" /><log4j:data name="log4net:HostName" value="webserver2" /></log4j:properties><log4j:locationInfo class="SomeOrganisation.Shared.ApplicationBlocks.Logging.Logger" method="Warn" file="E:\TFS2018agent\agent\_work\96\s\Shared\SomeOrganisation.Shared.ApplicationBlocks\Logging\Logging.cs" line="294" /></log4j:event>
我的logstash配置文件：
input {
file {
path => ["C:/Users/maskedUsername/Desktop/stackoverflow-log4net.txt"]
start_position => "beginning"
file_sort_by => "last_modified"
file_sort_direction => "desc"
sincedb_path => "NUL"
type => "appl"
codec => multiline {
pattern => "^<log4j:event"
negate => true
what => "previous"
}
}
}
filter {
if [type] == "appl" {
grok {
add_tag => [ "groked" ]
match => ["message", ".*"]
remove_tag => ["_grokparsefailure"]
}
xml {
source => "message"
remove_namespaces => true
target => "log4jevent"
xpath => [ "//event/@timestamp", "timestamp" ]
xpath => [ "//event/@level", "loglevel" ]
xpath => [ "/event/message/text()", "message" ]
xpath => [ "/event/throwable/text()", "exception" ]
xpath => [ "//event/properties/data[@name='log4jmachinename']/@value", "machinename" ]
xpath => [ "//event/properties/data[@name='log4japp']/@value", "app" ]
xpath => [ "//event/properties/data[@name='log4net:UserName']/@value", "username" ]
xpath => [ "//event/properties/data[@name='log4net:Identity']/@value", "identity" ]
xpath => [ "//event/properties/data[@name='log4net:HostName']/@value", "hostname" ]
}
mutate {
remove_field => ["type", "tags", "message"]
}
date {
match => [ "timestamp","UNIX" ]
target => "@timestamp"
remove_field => ["timestamp"]
}
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "log4jevents"
document_type => "log4jevent"
}
stdout { codec => rubydebug }
}
我的弹性搜索文档数据：
{
"took" : 0,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 4,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "log4jevents",
"_type" : "log4jevent",
"_id" : "kACTLnIBpHd52XYqFAtE",
"_score" : 1.0,
"_source" : {
"timestamp" : [
"1530819710045"
],
"tags" : [
"_dateparsefailure"
],
"host" : "localhost",
"machinename" : [
"webserver1"
],
"identity" : [
"user1"
],
"username" : [
"SomeOrganization\\!svc-app-identity"
],
"@timestamp" : "2020-05-19T20:14:49.672Z",
"@version" : "1",
"hostname" : [
"webserver1"
],
"loglevel" : [
"WARN"
],
"app" : [
"/LM/W3SVC/2/ROOT-1-131752914805620482"
],
"path" : "C:/Users/MaskedUsername/Desktop/stackoverflow-log4net.txt"
}
},
{
"_index" : "log4jevents",
"_type" : "log4jevent",
"_id" : "jwCTLnIBpHd52XYqFAtE",
"_score" : 1.0,
"_source" : {
"timestamp" : [
"1530819760731"
],
"tags" : [
"_dateparsefailure"
],
"host" : "localhost",
"machinename" : [
"webserver1"
],
"identity" : [
"user1"
],
"username" : [
"SomeOrganization\\!svc-app-identity"
],
"@timestamp" : "2020-05-19T20:14:49.700Z",
"@version" : "1",
"hostname" : [
"webserver1"
],
"loglevel" : [
"ERROR"
],
"app" : [
"/LM/W3SVC/2/ROOT-1-131752914805620482"
],
"path" : "C:/Users/MaskedUsername/Desktop/stackoverflow-log4net.txt"
}
},
{
"_index" : "log4jevents",
"_type" : "log4jevent",
"_id" : "kQCTLnIBpHd52XYqFAtE",
"_score" : 1.0,
"_source" : {
"timestamp" : [
"1530824089499"
],
"tags" : [
"_dateparsefailure"
],
"host" : "localhost",
"machinename" : [
"webserver1"
],
"username" : [
"SomeOrganization\\!svc-app-identity"
],
"@timestamp" : "2020-05-19T20:14:49.738Z",
"@version" : "1",
"hostname" : [
"webserver1"
],
"loglevel" : [
"ERROR"
],
"app" : [
"/LM/W3SVC/2/ROOT-2-131752976869399121"
],
"path" : "C:/Users/MaskedUsername/Desktop/stackoverflow-log4net.txt",
"exception" : [
"System.Security.Authentication.AuthenticationException: We can�t find that username and/or password. If you are trying to register for the first time using your employer�s credentials, select the Create Your Profile link below. If you are having trouble accessing the site, feel free to call us at none-one-CARES in the United States or Canada, 0800 000 000 in the United Kingdom, or 0800 000 000 in Ireland.\n at SomeOrganization.Shared.BusinessLogic.Security.Login.Save() in c:\\Builds\\5\\mainline\\Main.SomeApplication\\Sources\\mainline\\Main\\Shared\\SomeOrganization.Shared.BusinessLogic\\Security\\Login.cs:line 547"
]
}
},
{
"_index" : "log4jevents",
"_type" : "log4jevent",
"_id" : "kgCTLnIBpHd52XYqFAvT",
"_score" : 1.0,
"_source" : {
"timestamp" : [
"1530819760856"
],
"tags" : [
"_dateparsefailure"
],
"host" : "localhost",
"machinename" : [
"webserver1"
],
"identity" : [
"user1"
],
"username" : [
"SomeOrganization\\!svc-app-identity"
],
"@timestamp" : "2020-05-19T20:14:49.732Z",
"@version" : "1",
"hostname" : [
"webserver1"
],
"loglevel" : [
"ERROR"
],
"app" : [
"/LM/W3SVC/2/ROOT-1-131752914805620482"
],
"path" : "C:/Users/MaskedUsername/Desktop/stackoverflow-log4net.txt",
"exception" : [
"SomeOrganization.SomeApplication.BusinessLogic.Security.SomeOrganizationSomeApplicationException: Exception of type 'SomeOrganization.SomeApplication.BusinessLogic.Security.SomeOrganizationSomeApplicationException' was thrown.\n at SomeOrganization.SomeApplication.BusinessLogic.PaymentAccount.Save() in c:\\Builds\\5\\mainline\\Main.SomeApplication\\Sources\\mainline\\Main\\SomeApplication\\SomeOrganization.SomeApplication.BusinessLogic\\PaymentAccount.cs:line 415\n at Csla.BusinessBase`1.Save(Boolean forceUpdate) in C:\\andre\\mainline\\SomeApplicationNewDevelopment\\CSLA\\Source-4.3.12\\Csla\\BusinessBase.cs:line 163\n at Csla.BusinessBase`1.Csla.Core.ISavable.Save(Boolean forceUpdate) in C:\\andre\\mainline\\SomeApplicationNewDevelopment\\CSLA\\Source-4.3.12\\Csla\\BusinessBase.cs:line 350\n at SomeOrganization.Shared.Web.ApplicationBlocks.Controllers.CustomCslaMvcController.SaveObject[T](T item, Action`1 updateModel, Boolean forceUpdate) in c:\\Builds\\5\\mainline\\Main.SomeApplication\\Sources\\mainline\\Main\\Shared\\SomeOrganization.Shared.Web.ApplicationBlocks\\Controllers\\CustomCslaMvcController.cs:line 171"
]
}
}
]
}
}
我在这里想念什么？
解决方案
该UNIX模式期望您的时间是自纪元以来的 unix 时间（以秒为单位），这是一个 10 位数字，但您的timestamp字段是自纪元以来的以毫秒为单位的 unix 时间，一个 13 位数字。
您应该改用该UNIX_MS模式。
date {
match => [ "timestamp","UNIX_MS" ]
target => "@timestamp"
remove_field => ["timestamp"]
}
编辑：
如果字段是整数，则上述过滤器工作正常timestamp，但似乎xml过滤器将数据存储在数组中，即使您只有一个信息，所以timestamp这种情况下的字段在索引 0 中，然后是过滤器中的字段需要是[timestamp][0]
date {
match => [ "[timestamp][0]","UNIX_MS" ]
target => "@timestamp"
remove_field => ["timestamp"]
}
使用以下消息进行模拟。
{ "msg": "sample message", "timestamp": ["1530819710045"] }
输出是：
{
"@timestamp" => 2018-07-05T19:41:50.045Z,
"host" => "elk",
"@version" => "1",
"msg" => "sample message"
}
推荐阅读
- javascript - Axios 未在请求中发送自定义标头（可能是 CORS 问题）
- qaf - QMetry：com.qmetry.qaf.automation.step.client.ScenarioFactory.getTestsFromFile() 抛出异常
- sql - 动态透视 3 列表
- dynamics-crm - Dynamics 自定义操作流程会话
- python - Pandas 'reduce' 和 'accumulate' 功能 - 不完整的实现
- c# - 浏览 Javascript 生成的网页
- javascript - 如何在js中用字符串进行大写？
- excel - 再次与嵌套循环 VBA 混淆
- ios - 根据json中的值进行搜索
- multithreading - Qt。如何正确打印来自 QThread 对象的数据？