amazon-web-services - Cloudtrail 将日志记录到 AWS Elasticsearch

Question

尝试将多个 AWS 帐户的 cloudtrail 日志从 s3 获取到 elasticsearch 中，直到现在一切都停止了。错误显示如下所示

[2018-10-16T21:33:42,096][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error_message=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will_retry_in_seconds=>8}
[2018-10-16T21:33:44,406][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://vpc-sec-dummytext.eu-west-1.es.amazonaws.com:443/, :path=>"/"}
[2018-10-16T21:33:44,430][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"https://vpc-sec-dummytext.eu-west-1.es.amazonaws.com:443/"}
[2018-10-16T21:33:51,426][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>413, :url=>"https://vpc-sec-dummytext.eu-west-1.es.amazonaws.com:443/_bulk"}

这也是我的 logstash 配置，因为我正在使用 logstash 进行摄取

```

    input {
       s3 {
         bucket => "dummy-s3"
         region => "eu-west-1"
         type => "cloudtrail"
         sincedb_path => "/tmp/logstash/cloudtrail"
         exclude_pattern => "/CloudTrail-Digest/"
         interval => 120
         codec => "json"
       }
    }

    filter {
        if [type] == "cloudtrail" {
            json {
                source => "message"
            }
            split {
                    field => "Records"
                    add_tag => "splitted"

            }
            if ("splitted" in [tags]) {
              date {
                    match => ["eventTime", "ISO8601"]
                    remove_tag => ["splitted"]
                    remove_field => ["timestamp"]
                   }
            }

            geoip {
                source => "[Records][sourceIPAddress]"
                target => "geoip"
                add_tag => ["cloudtrail-geoip"]
            }

            mutate {
                 gsub => [

                  "eventSource", "\.amazonaws\.com$", "",
                  "apiVersion", "_", "-"
            ]

            }
        }
    }

    output {
        elasticsearch {
          hosts => ["vpc-sec-dummytext.eu-west-1.es.amazonaws.com:443"]
          ssl => true
          index => "cloudtrail-%{+YYYY.MM.dd}"
          doc_as_upsert => true
          template_overwrite => true
          }
        stdout {
         codec => rubydebug
        }
     }
}

当 log-stash 从 ubuntu ec2 启动或重新启动时，日志被摄取几分钟然后停止

任何帮助将不胜感激。

标签： amazon-web-serviceselasticsearchlogstashkibana-6amazon-cloudtrail