时间戳

首页 > 解决方案 > Elasticsearch-hadoop 连接器

scala - Elasticsearch-hadoop 连接器

问题描述

我想使用 SSL 协议通过 elasticsearch-hadoop 连接到 Elasticsearch [7.5] 集群。通过 curl 提供用户和密码：

curl -XPUT -k -u "user:pass" https://xxxxxxxxxxx.com:9200/prefix-test
curl -XGET -k -u "user:pass" https://xxxxxxxxxxx.com:9200/prefix-test

我能够创建索引并获得响应：

{"prefix-test":{"aliases":{},"mappings":{},"settings":{"index":{"creation_date":"1595854154818","number_of_shards":"1","number_of_replicas":"1","uuid":"k7Djh_fgHGIUTG76868Tuygi","version":{"created":"7050099"},"provided_name":"prefix-test"}}}}

这里“用户”只能读取或写入以“前缀”为前缀的索引。

但是，当我尝试在 Scala 中使用 elasticsearch-hadoop [7.5] 和 Spark [2.3.1] 连接到该集群时，配置如下：

"es.nodes": "xxxxxxxxxx.com"
"es.port": "9200",
"es.net.http.auth.user": "user"
"es.net.http.auth.pass": "password"
"es.net.ssl": "true"
"es.index.auto.create": "yes"
"es.net.ssl.cert.allow.self.signed": "true"
"es.nodes.path.prefix": "/prefix"

然后我尝试插入数据：

.saveToEs("prefix-test")

这给了我：

org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest:org.elasticsearch.hadoop.rest.EsHadoopRemoteException: index_not_found_exception: no such index [prefix]

并且没有参数 "es.nodes.path.prefix": "/prefix" ：

org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest:org.elasticsearch.hadoop.rest.EsHadoopRemoteException: security_exception: action [cluster:monitor/main] is unauthorized for user [user]

我在这里缺少什么？如何使用这个前缀参数？

可能相关：https ://discuss.elastic.co/t/spark-job-is-failing-with-authenticating-with-basic-error/25732/2

标签： scalaapache-sparkelasticsearchhadoop

解决方案

推荐阅读