logstash - logstash 解析 apache drupal 日志
问题描述
我有这个日志:
[Fri May 15 01:20:48.661420 2020] [proxy_fcgi:error] [pid 11943:tid 139947462346496] [client 154.15.19.140:64717] AH01071: Got error 'PHP message: Company Care|https://webdemo2.nics.com|severity=error|type=feature_card_manager|ip=154.15.19.140|uri=https://webdemo2nics.com/ca/user/payment-information|referer=https://webdemo2.nics.com/ca/user/payment-information|uid=68055|link=|message=Failed to add card. HTTP Response: {"code":71,"category":1,"message":"Declined","reference":""}', referer: https://webdemo2.nics.com/ca/user/payment-information
我正在尝试通过logstash发送它。这是我的logstash配置:
input {
file {
path => "/tmp/test.log"
}
}
filter {
date {
match => [ "logdate", "E MMM dd HH:mm:ss.SSS yyyy" ]
target => "@timestamp"
}
grok {
match => { "message" => "\[(?<timestamp>%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR})\]\s+\[%{WORD}:%{WORD:loglevel}\].*\[client %{URIHOST:client}\]\s++%{WORD:error_code}\:.*\|type=%{WORD:type}\|.*\|uri=%{URIPATH:uri}\|uid=%{NUMBER:uid}\|.*%{GREEDYDATA:message}$"
}
overwrite => ["message"]
}
mutate{
remove_field => "@version"
}
}
output {
stdout {
codec => rubydebug
}
}
问题是它没有用时间戳覆盖@timestamp。我尝试使用 mutate rename 但这只是让我失去了这个领域。有谁知道如何更改配置文件来解决时间戳问题?此外,时间戳似乎被解析为字符串而不是日期对象。它应该来自我的日期模式还是日期模式和 grok 模式的组合?
这是输出:
{
"client" => "154.15.19.140:64717",
"error_code" => "AH01071",
"@timestamp" => 2020-07-15T16:58:52.916Z,
"loglevel" => "error",
"host" => "kourosh-VirtualBox",
"port" => "64717",
"path" => "/tmp/test.log",
"timestamp" => "Fri May 15 01:20:48.661420 2020",
"message" => "[Fri May 15 01:20:48.661420 2020] [proxy_fcgi:error] [pid 11943:tid 139947462346496] [client 154.15.19.140:64717] AH01071: Got error 'PHP message: Company|https://webdemo2.nics.com|severity=error|type=feature_card_manager|ip=154.5.59.140|uri=https://webdemo2.nics.com/ca/user/payment-information|referer=https://webdemo2.nics.com/ca/user/payment-information|uid=55|link=|message=Failed to add card. HTTP Response: {\"code\":71,\"category\":1,\"message\":\"Declined\",\"reference\":\"\"}', referer: https://webdemo2.nics.com/ca/user/payment-information"
}
解决方案
我想到了。date 部分需要在 grok 部分之后,并且 target 可用于覆盖 @timestamp。这是工作配置:
input {
file {
path => "/tmp/test.log"
}
}
filter {
grok {
match => { "message" => "\[(?<event_timestamp>%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR})\]\s+\[%{WORD}:%{WORD:loglevel}\].*\[client %{URIHOST:client}\]\s+%{WORD:error_code}\:.*\|type=%{WORD:type}\|.*\|uri\=%{GREEDYDATA:uri}\|referer.*\|uid=%{NUMBER:uid}\|.*\|message\=%{GREEDYDATA:message}(\s+HTTP Response\:%{GREEDYDATA:response}.*)?(\,\sreferer.*).*"
}
overwrite => ["message"]
}
date {
match => [ "event_timestamp", "E MMM dd HH:mm:ss.SSSSSS yyyy" ]
target => "@timestamp"
add_field => { "debug" => "timestampMatched"}
}
mutate{
remove_field => ["@version", "@tags"]
}
}
output {
stdout {
codec => rubydebug
}
}
推荐阅读
- flutter - 平台通道代码的颤振隔离问题
- java - Generating csv file for multiple loop in Java?
- java - 我无法使用 Spring Boot 从 App Engine 中的服务连接到 Google Datastore
- git - Github 错误添加 https 来源,但它给了我错误 ---verbose
- php - 如果页面出现 504 错误,fgets 不会终止。有人知道如何为 fgets 设置超时吗
- php - 不发送数据不显示取自数据库的数组
- ubuntu - 从nginx中的url中提取第一个点之前的单词
- c# - 如何处理revit api中管道配件的csv文件中某些行的问题?
- python - Tkinter 中的错误几何说明符
- c++ - 如何在 C++ 中有效地进行字符串连接?