首页 > 解决方案 > geoip.location 被定义为映射 [doc] 中的对象,但此名称已用于其他类型的字段

问题描述

我收到此错误:

无法将事件索引到 Elasticsearch。{:status=>400, :action=>["index", {:_id=>nil, :_index=>"nginx-access-2018-06-15", :_type=>"doc", :_routing= >nil}, #], :response=>{"index"=>{"_index"=>"nginx-access-2018-06-15", "_type"=>"doc", "_id"=>" jo-rfGQBDK_ao1ZhmI8B", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"[geoip.location] 在映射 [doc] 中被定义为一个对象,但这名称已用于其他类型的字段"}}}}

我收到上述错误,但不明白为什么,这是加载到一个没有数据的全新 ES 实例中。这是插入的第一条记录。为什么我会收到此错误?这是配置:

input {

  file {
    type => "nginx-access"
    start_position => "beginning"
    path => [ "/var/log/nginx-archived/access.log.small"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}

filter {
  if [type] == "nginx-access" {
       grok {
         patterns_dir => "/etc/logstash/patterns"
         match => { "message" => "%{NGINX_ACCESS}" }
         remove_tag => ["_grokparsefailure"]
       }
       geoip {
         source => "visitor_ip"
       }
       date {
         # 11/Jun/2018:06:23:45 +0000
         match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
         target => "@request_time"
       }
       if "_grokparsefailure" not in [tags] {
            ruby {
              code => "
                thetime = event.get('@request_time').time
                event.set('index_date', 'nginx-access-' + thetime.strftime('%Y-%m-%d'))
              "
            }
          }
         if "_grokparsefailure" in [tags] {
              ruby {
                code => "
                  event.set('index_date', 'nginx-access-error')
                "
              }
            }
     }

}

output {
  elasticsearch {
    hosts => "elasticsearch:9200"
    index => "%{index_date}"
    template => "/etc/logstash/templates/nginx-access.json"
    template_overwrite => true
    manage_template => true
    template_name => "nginx-access"
  }
  stdout { }
}

这是一个示例记录:

{
            "method" => "GET",
          "@version" => "1",
             "geoip" => {
         "continent_code" => "AS",
               "latitude" => 39.9289,
           "country_name" => "China",
                     "ip" => "220.181.108.103",
               "location" => {
             "lon" => 116.3883,
             "lat" => 39.9289
         },
            "region_code" => "11",
            "region_name" => "Beijing",
              "longitude" => 116.3883,
               "timezone" => "Asia/Shanghai",
              "city_name" => "Beijing",
          "country_code2" => "CN",
          "country_code3" => "CN"
     },
        "index_date" => "nginx-access-2018-06-15",
            "ignore" => "\"-\"",
             "bytes" => "2723",
           "request" => "/wp-login.php",
     "@request_time" => 2018-06-15T06:29:40.000Z,
           "message" => "220.181.108.103 - - [15/Jun/2018:06:29:40 +0000] \"GET /wp-login.php HTTP/1.1\" 200 2723 \"-\" \"Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)\"",
              "path" => "/var/log/nginx-archived/access.log.small",
        "@timestamp" => 2018-07-09T01:32:56.952Z,
              "host" => "ab1526efddec",
        "visitor_ip" => "220.181.108.103",
         "timestamp" => "15/Jun/2018:06:29:40 +0000",
          "response" => "200",
          "referrer" => "\"Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)\"",
       "httpversion" => "1.1",
              "type" => "nginx-access"
 }

标签: elasticsearchlogstash

解决方案


想出答案,基于此:

https://www.elastic.co/guide/en/elasticsearch/reference/6.x/removal-of-types.html#_schedule_for_removal_of_mapping_types

基本问题是,对于每个 Elasticsearch 索引,每个字段必须是相同类型,即使记录是不同类型。

也就是说,如果我将一个人{ "status": "A" }存储为文本,我就无法将汽车的记录{ "status": 23 }存储为同一索引中的数字。根据上面链接中的信息,我为每个索引存储一个“类型”。

我的 Logstash 输出部分如下所示:

output {
  elasticsearch {
    hosts => "elasticsearch:9200"
    index => "%{index_date}"
    # Can test loading this with:
    # curl -XPUT -H 'Content-Type: application/json' -d@/docker-elk/logstash/templates/nginx-access.json http://localhost:9200/_template/nginx-access
    template => "/etc/logstash/templates/nginx-access.json"
    template_overwrite => true
    manage_template => true
    template_name => "nginx-access"
  }
  stdout { }
}

我的模板如下所示:

{
  "index_patterns": ["nginx-access*"],
  "settings": {

  },
  "mappings": {
    "doc": {
      "_source": {
        "enabled": true
      },
      "properties": {
    "type" : { "type": "keyword" },
    "response_time": { "type": "float" },
        "geoip" : {
      "properties" : {
        "location": {
              "type": "geo_point"
            }
          }
    }
      }
    }
  }
}

我还使用上面链接中描述的每个索引模式的一种类型。


推荐阅读