时间戳

首页 > 解决方案 > Logstash 无法连接到部署在 Kubernetes 集群上的 Elasticsearch

elasticsearch - Logstash 无法连接到部署在 Kubernetes 集群上的 Elasticsearch

问题描述

我已经在 EKS 集群上部署了 Logstash 和 elasticsearch pod。当我检查 logstash pod 的日志时，它显示无法访问 elasticserach 服务器。虽然我的弹性搜索已经启动并正在运行。请在下面找到 yaml 文件并记录错误。

configMap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: "logstash-configmap-development"
  namespace: "development"
  labels:
    app: "logstash-development"
data:
  logstash.conf: |-
    input {
         http {
         
      }
    }
    
    filter {
        json {
            source => "message"
        }
    }
    output {
      elasticsearch {
            hosts => ["https://my-server.com/elasticsearch-development/"]
            index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
        }
        
        stdout {
            codec => rubydebug
      }
    }

部署.yaml

---
  apiVersion: "apps/v1"
  kind: "Deployment"
  metadata: 
    name: "logstash-development"
    namespace: "development"
  spec: 
    selector: 
      matchLabels: 
        app: "logstash-development"
    replicas: 1
    strategy: 
      type: "RollingUpdate"
      rollingUpdate: 
        maxSurge: 1
        maxUnavailable: 1
    minReadySeconds: 5
    template: 
      metadata: 
        labels: 
          app: "logstash-development"
      spec: 
        containers: 
          - 
            name: "logstash-development"
            image: "logstash:7.10.2"
            imagePullPolicy: "Always"
            env: 
              - 
                name: "XPACK_MONITORING_ELASTICSEARCH_HOSTS"
                value: "https://my-server.com/elasticsearch-development/"
                
              -
                name: "XPACK_MONITORING_ELASTICSEARCH_URL"
                value: "https://my-server.com/elasticsearch-development/"
                
              -
              
                name: "SERVER_BASEPATH"
                value: "logstash-development"
                
                
                
            securityContext:
                privileged: true
                
            ports: 
              - 
                containerPort: 8080
                protocol: TCP
                
            volumeMounts:
                -
                  name: "logstash-conf-volume"
                  mountPath: "/usr/share/logstash/pipeline/"
                  
        volumes:
            -
              name: "logstash-conf-volume"
              configMap:
                 name: "logstash-configmap-development"
                 items:
                    - key: "logstash.conf"
                      path: "logstash.conf"
        imagePullSecrets: 
          - 
            name: "logstash"

服务.yaml

---
  apiVersion: "v1"
  kind: "Service"
  metadata: 
    name: "logstash-development"
    namespace: "development"
    labels: 
      app: "logstash-development"
  spec: 
    ports: 
      - 
        port: 55770
        targetPort: 8080
    selector: 
      app: "logstash-development"

Logstash pod 日志错误

[2021-06-09T08:22:38,708][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://my-server.com/elasticsearch-development/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://my-server.com/elasticsearch-development/][Manticore::ConnectTimeout] connect timed out"}

注意：- Elasticsearch 已启动并正在运行。当我点击logstash url时，它给出的状态是好的。

我检查了elasticsearch cluster-ip，他们的logstash 能够与Elasticsearch 连接，但是当我提供入口路径url 时，它无法连接到elasticsearch。

同样从日志中，我注意到它为 elasticsearch 使用了不正确的 url。

我的弹性搜索网址是这样的：- https://my-server.com/elasticserach

但是logstash正在寻找https://my-server.com:9200/elasticsearch

使用此 url ( https://my-server.com:9200/elasticsearch ) 无法访问 elasticsearch，因为它会导致连接超时。

有人能说出为什么要使用（https://my-server.com:9200/elasticsearch）而不是（https://my-server.com/elasticsearch）

标签： elasticsearchkubernetesconnectionlogstash

解决方案

我现在可以将logstash与elasticsearch连接起来，如果您使用带有dns名称的elasticsearch，logstash默认情况下会将elasticsearch的端口作为9200，所以在我的情况下，它将elasticsearch url作为https://my-server。 com:9200/elasticsearch-development/。但是使用该 url，elasticsearch 无法访问，只能通过 ( https://myserver.com/elasticsearch-development/ ) 访问。所以我需要在我的elasticsearch url中添加https端口，即443，logstash可以通过它连接到elasticserach（https://my-server.com:443/elasticsearch-development/）

长话短说：-

在环境变量 XPACK_MONITORING_ELASTICSEARCH_HOSTS 和 XPACK_MONITORING_ELASTICSEARCH_URL 下的 deployment.yaml 文件中，给定值为https://my-server.com:443/elasticsearch-development/

logstash.conf 文件中给出了相同的值。

推荐阅读