elasticsearch - geoip.location 被定义为映射 [doc] 中的对象,但此名称已用于其他类型的字段
问题描述
我收到此错误:
无法将事件索引到 Elasticsearch。{:status=>400, :action=>["index", {:_id=>nil, :_index=>"nginx-access-2018-06-15", :_type=>"doc", :_routing= >nil}, #], :response=>{"index"=>{"_index"=>"nginx-access-2018-06-15", "_type"=>"doc", "_id"=>" jo-rfGQBDK_ao1ZhmI8B", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"[geoip.location] 在映射 [doc] 中被定义为一个对象,但这名称已用于其他类型的字段"}}}}
我收到上述错误,但不明白为什么,这是加载到一个没有数据的全新 ES 实例中。这是插入的第一条记录。为什么我会收到此错误?这是配置:
input {
file {
type => "nginx-access"
start_position => "beginning"
path => [ "/var/log/nginx-archived/access.log.small"]
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
if [type] == "nginx-access" {
grok {
patterns_dir => "/etc/logstash/patterns"
match => { "message" => "%{NGINX_ACCESS}" }
remove_tag => ["_grokparsefailure"]
}
geoip {
source => "visitor_ip"
}
date {
# 11/Jun/2018:06:23:45 +0000
match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
target => "@request_time"
}
if "_grokparsefailure" not in [tags] {
ruby {
code => "
thetime = event.get('@request_time').time
event.set('index_date', 'nginx-access-' + thetime.strftime('%Y-%m-%d'))
"
}
}
if "_grokparsefailure" in [tags] {
ruby {
code => "
event.set('index_date', 'nginx-access-error')
"
}
}
}
}
output {
elasticsearch {
hosts => "elasticsearch:9200"
index => "%{index_date}"
template => "/etc/logstash/templates/nginx-access.json"
template_overwrite => true
manage_template => true
template_name => "nginx-access"
}
stdout { }
}
这是一个示例记录:
{
"method" => "GET",
"@version" => "1",
"geoip" => {
"continent_code" => "AS",
"latitude" => 39.9289,
"country_name" => "China",
"ip" => "220.181.108.103",
"location" => {
"lon" => 116.3883,
"lat" => 39.9289
},
"region_code" => "11",
"region_name" => "Beijing",
"longitude" => 116.3883,
"timezone" => "Asia/Shanghai",
"city_name" => "Beijing",
"country_code2" => "CN",
"country_code3" => "CN"
},
"index_date" => "nginx-access-2018-06-15",
"ignore" => "\"-\"",
"bytes" => "2723",
"request" => "/wp-login.php",
"@request_time" => 2018-06-15T06:29:40.000Z,
"message" => "220.181.108.103 - - [15/Jun/2018:06:29:40 +0000] \"GET /wp-login.php HTTP/1.1\" 200 2723 \"-\" \"Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)\"",
"path" => "/var/log/nginx-archived/access.log.small",
"@timestamp" => 2018-07-09T01:32:56.952Z,
"host" => "ab1526efddec",
"visitor_ip" => "220.181.108.103",
"timestamp" => "15/Jun/2018:06:29:40 +0000",
"response" => "200",
"referrer" => "\"Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)\"",
"httpversion" => "1.1",
"type" => "nginx-access"
}
解决方案
想出答案,基于此:
基本问题是,对于每个 Elasticsearch 索引,每个字段必须是相同类型,即使记录是不同类型。
也就是说,如果我将一个人{ "status": "A" }
存储为文本,我就无法将汽车的记录{ "status": 23 }
存储为同一索引中的数字。根据上面链接中的信息,我为每个索引存储一个“类型”。
我的 Logstash 输出部分如下所示:
output {
elasticsearch {
hosts => "elasticsearch:9200"
index => "%{index_date}"
# Can test loading this with:
# curl -XPUT -H 'Content-Type: application/json' -d@/docker-elk/logstash/templates/nginx-access.json http://localhost:9200/_template/nginx-access
template => "/etc/logstash/templates/nginx-access.json"
template_overwrite => true
manage_template => true
template_name => "nginx-access"
}
stdout { }
}
我的模板如下所示:
{
"index_patterns": ["nginx-access*"],
"settings": {
},
"mappings": {
"doc": {
"_source": {
"enabled": true
},
"properties": {
"type" : { "type": "keyword" },
"response_time": { "type": "float" },
"geoip" : {
"properties" : {
"location": {
"type": "geo_point"
}
}
}
}
}
}
}
我还使用上面链接中描述的每个索引模式的一种类型。
推荐阅读
- php - Codeigniter 检查编辑功能上的重复项
- php - PHP:AdjacentElementsProduct - CodeFights https://app.codesignal.com/arcade/intro/level-2/
- c# - 制作、保存和显示 bmp
- sql - 动态地将日期行旋转到列标题中,并为每个日期重复未透视的列
- google-docs-api - 我希望能够使用 API 以编程方式将 Google 演示文稿中的幻灯片插入(链接)到 Google 文档中
- scala - 解析搜索表达式上的括号 - Scala 解析器组合器
- python - 无法将寡妇时代时间转换为正常日期时间
- django - 我希望我的 django 应用程序从远程机器获取文件
- php - PHP 不显示在 HTML 页面上
- scala - SparkException:无法从 JAR 文件加载主类:/root/master