时间戳

首页 > 解决方案 > Elasticsearch 中的排序导致一致性问题

elasticsearch - Elasticsearch 中的排序导致一致性问题

问题描述

我正在将我的数据发送到带有 index_number 文档的 elasticsearch。它的唯一标识符。当我尝试用它对其进行排序时，从 python 客户端我得到了这个一致性问题，如图所示。

索引号

这是我的查询 dsl

"size": 1,
"query": {
    "match_all": {}
},
"sort": [
    {
        "index_number.keyword": {
            "order": "asc",
            "missing": "_last",
            "unmapped_type": "String"
        }
    }
]

在logstash输出中

output{
    elasticsearch {
        hosts => ["localhost:9200"]
        index => "logstash_%{+yyyy-MM-dd}"
        manage_template => true
        template_name => "logstash_template"
        template => "..../logstash_template.json"
        http_compression => true
    }
}

In my logstash template.json
...

{
"index_patterns": ["logstash_*"],
"template": {
  "settings":{
    "number_of_shards": 1,
    "number_of_replicas": 0,
    "index": {
      "sort.field": "index_number", 
      "sort.order": "asc"  
    }
  },
  "mappings": {
    "dynamic_templates":{
      "string_fields": {
        "match": "*",
        "match_mapping_type": "string",
        "mapping": {"type":"keyword"}
      }
    },
    "properties": {
      "index_number": {
        "type": "keyword",
        "fields": {
          "numeric": {
            "type": "double"
          }
        }
      }
    }
  }
}
  }

....

elasticsearch上的映射

{
  "logstash_2020-03-12" : {
    "mappings" : {
      "properties" : {
        .....
        "index_number" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "city" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "country" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        -----
      }
    }
  }
}

我该如何解决？谢谢回答。

标签： elasticsearch

解决方案

您需要添加template_overwrite到 Logstash 输出配置，否则logstash_template如果它已经存在则不会被覆盖：

output{
    elasticsearch {
        hosts => ["localhost:9200"]
        index => "logstash_%{+yyyy-MM-dd}"
        manage_template => true
        template_override => true                         <-- add this
        template_name => "logstash_template"
        template => "..../logstash_template.json"
        http_compression => true
    }
}

确保您的logstash_template.json文件具有以下格式：

{
  "index_patterns": [
    "logstash_*"
  ],
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 0,
    "index": {
      "sort.field": "index_number",
      "sort.order": "asc"
    }
  },
  "mappings": {
    "dynamic_templates": {
      "string_fields": {
        "match": "*",
        "match_mapping_type": "string",
        "mapping": {
          "type": "keyword"
        }
      }
    },
    "properties": {
      "index_number": {
        "type": "keyword",
        "fields": {
          "numeric": {
            "type": "double"
          }
        }
      }
    }
  }
}

您已经包含在该部分中mappings，但这仅适用于 Logstash 输出尚不支持的新索引模板。您需要使用旧版索引模板。settingstemplateelasticsearch

推荐阅读