elasticsearch - Logstash 拆分过滤器
问题描述
最近我发现我可以通过直接提供 URL 直接从 Logstash 中汇集数据。获取输入效果很好,但是它将完整的文档下载并加载到 ES 中。
我想为每一行创建一个弹性搜索的新记录。默认情况下,整个文件加载在消息字段中,它会减慢 Kibana 在 Discovery 选项卡等中的加载速度。
Kibana 输出:
{
"_index": "blacklists",
"_type": "default",
"_id": "pf3k_2QB9sEBYW4CK4AA",
"_version": 1,
"_score": null,
"_source": {
"@timestamp": "2018-08-03T13:05:00.569Z",
"tags": [
"_jsonparsefailure",
"c2_info",
"ipaddress"
],
"@version": "1",
"message": "#############################################################\n## Master Feed of known, active and non-sinkholed C&Cs IP \n## addresses\n## \n## HIGH-CONFIDENCE FAMILIES ONLY\n## \n## Feed generated at: 2018-08-03 12:13 \n##\n## Feed Provided By: John Bambenek of Bambenek Consulting\n## jcb@bambenekconsulting.com // http://bambenekconsulting.com\n## Use of this feed is governed by the license here: \n## http://osint.bambenekconsulting.com/license.txt,
"client": "204.11.56.48",
"http_poller_metadata": {
"name": "bembenek_c2",
"host": "node1",
"request": {
"method": "get",
"url": "http://osint.bambenekconsulting.com/feeds/c2-ipmasterlist-high.txt"
},
"response_message": "OK",
"runtime_seconds": 0.27404,
"response_headers": {
"content-type": "text/plain",
"accept-ranges": "bytes",
"cf-ray": "4448fe69e02197ce-FRA",
"date": "Fri, 03 Aug 2018 13:05:05 GMT",
"connection": "keep-alive",
"last-modified": "Fri, 03 Aug 2018 12:13:44 GMT",
"server": "cloudflare",
"vary": "Accept-Encoding",
"etag": "\"4bac-57286dbe759e4-gzip\""
},
"code": 200,
"times_retried": 0
}
},
"fields": {
"@timestamp": [
"2018-08-03T13:05:00.569Z"
]
},
"sort": [
1533301500569
]
}
Logstash 配置:
input {
http_poller {
urls => {
bembenek_c2 => "http://osint.bambenekconsulting.com/feeds/c2-ipmasterlist-high.txt"
bembenek_c2dom => "http://osint.bambenekconsulting.com/feeds/c2-dommasterlist-high.txt"
blocklists_all => "http://lists.blocklist.de/lists/all.txt"
}
request_timeout => 30
codec => "json"
tags => c2_info
schedule => { cron => "*/10 * * * *"}
metadata_target => "http_poller_metadata"
}
}
filter {
grok {
match => { "message" => [
"%{IPV4:ipaddress}" }
add_tag => [ "ipaddress" ]
}
}
output {
stdout { codec => dots }
elasticsearch {
hosts => ["10.0.50.51:9200"]
index => "blacklists"
document_type => "default"
template_overwrite => true
}
file {
path => "/tmp/blacklists.json"
codec => json {}
}
}
有谁知道如何用“\n”分割加载的文件?
我试过了
filter {
split {
terminator => "\n"
}
}
如何使用此过滤器的文档和示例并不那么受欢迎。
解决方案
缺少的过滤器是:
filter {
split {
field => "[message]"
}
}
我们不必指定终止符,因为根据 Logstash 6.3 文档,它默认设置为“\n”。
推荐阅读
- javascript - 如何使用 React 在对话框(material-ui)中传递组件?
- azure - 我可以使用 Azure MFA 而不使用其 SSO 登录吗?
- python - 打印 0 之前的最后一个结果
- c# - 更新/删除列表视图中包含倒数计时器的行
- javascript - 如何自动复制输入的文本?
- javascript - Nuxt Axios 在页面之间移动时返回错误
- mysql - MySQL 如果行重复,则首选匹配字段值的行
- typescript - 有没有办法只为第二个泛型而不是第一个在打字稿中具有默认值的泛型传递值?
- html - 如何垂直对齐表单中的输入字段?
- c++ - 我的电脑无法编译数据类型:__int128