elasticsearch - Filebeat decode_json_fields 不解析数组
问题描述
我们正在使用 filbeat 处理器 decode_json-fields 来处理 Json 中的日志消息。我们遇到的问题是我们的一些日志是多层的,有很多数组和一些嵌套对象。我们尝试使用将 process_array 标志设置为 true 的 decode_json_fields,但 Filebeat 仍将“[”后面的所有内容放在一个字段中。
这是我们在 Kibana 的 Display 上得到的:
@timestamp Oct 28, 2019 @ 12:22:06.610
t _id pKEaEm4B7zyLz8s9M8Xe
t _index filebeat-7.3.2-2019.10.28-000001
# _score -
t _type _doc
t agent.ephemeral_id 7c3cd7b7-2f76-424e-a417-5aa82f119bed
t agent.hostname ******
t agent.id 571154fa-e864-49b1-a224-9d405befeddf
t agent.type filebeat
t agent.version 7.3.2
? circuitPath { "policy": "Health Check LB", "execTime": 0, "filters": [ { "class": "com.vordel.circuit.attribute.CompareAttributeFilter", "status": "Pass", "filterTime": 1557733771853, "execTime": 0, "espk": "PrimaryStore-43595d15-05f6-4135-aa9a-e8b9b2a35bda:-439438454261778670", "name": "Compare Attribute", "type": "CompareAttributeFilter" }, { "execTime": 0, "espk": "PrimaryStore-43595d15-05f6-4135-aa9a-e8b9b2a35bda:-6704867506249825459", "name": "Set Message - OK", "type": "ChangeMessageFilter", "class": "com.vordel.circuit.conversion.ChangeMessageFilter", "status": "Pass", "filterTime": 1557733771853 }, { "execTime": 0, "espk": "PrimaryStore-43595d15-05f6-4135-aa9a-e8b9b2a35bda:-5308572925601299001", "name": "Reflect - OK", "type": "ReflectFilter", "class": "com.vordel.circuit.net.ReflectFilter", "status": "Pass", "filterTime": 1557733771853 } ] }
? correlationId *******************
t ecs.version 1.0.1
t host.name *****
t input.type log
t log.file.path *****
# log.offset 747,788
? processInfo.domainId *******************
? processInfo.groupId group-2
? processInfo.groupName ******
? processInfo.hostname f3slsea310
? processInfo.serviceId instance-6
? processInfo.serviceName ******
? processInfo.version 7.6.2 SP1
suricata.eve.timestamp Oct 28, 2019 @ 12:22:06.610
? timestamp 1557733771854
如您所见,Filebeat 到达嵌套数组“circuitPath”的那一刻,它会解析单个字段中的所有内容,直到数组关闭。
这是我们遇到问题的日志之一的示例:
{"timestamp":1557733646862,"correlationId":"***************","processInfo":{"hostname":"f3slsea310","domainId":"*******************","groupId":"group-2","groupName":"*****","serviceId":"instance-6","serviceName":"*******","version":"7.6.2 SP1"},"circuitPath":[ { "policy": "Health Check LB", "execTime": 0, "filters": [ { "espk": "PrimaryStore-43595d15-05f6-4135-aa9a-e8b9b2a35bda:-439438454261778670", "name": "Compare Attribute", "type": "CompareAttributeFilter", "class": "com.vordel.circuit.attribute.CompareAttributeFilter", "status": "Pass", "filterTime": 1557733646861, "execTime": 0 } , { "espk": "PrimaryStore-43595d15-05f6-4135-aa9a-e8b9b2a35bda:-6704867506249825459", "name": "Set Message - OK", "type": "ChangeMessageFilter", "class": "com.vordel.circuit.conversion.ChangeMessageFilter", "status": "Pass", "filterTime": 1557733646861, "execTime": 0 } , { "espk": "PrimaryStore-43595d15-05f6-4135-aa9a-e8b9b2a35bda:-5308572925601299001", "name": "Reflect - OK", "type": "ReflectFilter", "class": "com.vordel.circuit.net.ReflectFilter", "status": "Pass", "filterTime": 1557733646861, "execTime": 0 } ] } ]}
文件节拍.yml
processors:
- decode_json_fields:
fields: [message]
process_array: true
max_depth: 11
overwrite_keys: true
解决方案
您必须指定“目标”指令,以便filebeat理解将值放在哪个字段中,或者将其留空 - “”以便所有这些值都到根
https://www.elastic.co/guide/en/beats/filebeat/current/decode-json-fields.html
推荐阅读
- python - 如何生成具有 99% 概率值落入 (3, 9) 排他性的随机值?
- python - Keras 后端函数,仅当值介于 a 和 b 之间时才给出
- python - 使用 df.head() 循环
- java - 我正在尝试将 java 对象发布到 .net 核心 web api。但我需要在 java 中发布对象数组
- javascript - 如何使用 Javascript 将 22:00:00 转换为 10 PM 或 22:00?
- python - mysql - 修复错误“TypeError: 'long' object is not iterable”
- azure - 是否有一种自动方法可以在关联任务完成时从 Azure Batch 池中删除节点?
- delphi - 基于INDY的Delphi SSL MITM Proxy - 内容加载问题
- python - 怎么能每隔一段时间发送文件?
- javascript - 如何使用代码而不是按钮单击弹出javascript模式