logstash - 如何在 Logstash 中为 Elasticsearch 服务过滤 AWS Lambda 和 API Gateway 日志？

Question

我在 S3 存储桶中有 AWS Lambda 和 API 网关日志，我托管了 logstash 并将日志从 s3 移动到 elasticsearch 服务以集中日志记录。我想在logstash中过滤Lambda和api网关的日志，所以在弹性搜索中我可以很容易地找到东西。

下面是我在 kibana“消息”字段中获取并想要过滤的 apigateway 日志

{"messageType":"DATA_MESSAGE",
"owner":"",
"logGroup":"API-Gateway-Execution-Logs/live",
"logStream":"d645920e395fe","subscriptionFilters":["API-Gateway-Execution-Logslive"],
"logEvents":[{"id":"3463","timestamp":1553,"message":"(d7b307ed-4c36-11e9) Extended Request Id: W6ghjgj"}]
}

filter i tried with
filter {
    grok {
        match => { "message" => "%{GREEDYDATA:wd}" }
    }
    json{
        source => "wd"
        target => "js"
    }
    mutate {
        add_field => { "t1" => "%{[js][logEvents][message]}"}
    }
}

标签： logstashaws-api-gatewayelastic-stacklogstash-grok

Answer 1

我已经使用 json 解析，使用 mutate https://www.elastic.co/guide/en/logstash/current/plugins-filters-split.html拆分和使用值

filter {
    json {
        source => "message"
    }
    split {
        field => "logEvents"
    }
    mutate {
        add_field => ["time", "%{[logEvents][timestamp]}"]
    }