apache-spark - 无法检测 ES 版本 - 如果网络/Elasticsearch 集群不可访问（HIVE），通常会发生这种情况

Question

我目前正在尝试仅执行从 Hive 到 ElasticSearch 的“SELECT * FROM table”。我正在使用 cloudera CDH 6.0.1。我已经将 elasticsearch-hadoop-hive-7.1.1 jar 添加到我的配置单元路径中。我有 ElasticSearch 7.1.1 Cloudera 堆栈和弹性运行在不同的服务器但在同一个网络中。

CREATE EXTERNAL TABLE ctrl_rater_resumen_lla_es  
( 
fecha_registro string, 
direccion string, 
linea_b_codigo_prestadora string, 
linea_b_tipo_numero string, 
es_roaming string,
call_duration string, 
linea_b_routing_number string, 
minutos string, fecha string 
) 
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES (
'es.resource' = 'ctrl_rater_resumen_lla/hb',
'es.node' = 'http://10.129.x.xxx',
'es.port' = '9200',
'es.index.auto.create' = 'true',
'es.index.read.missing.as.empty' = 'true',
'es.nodes.discovery'='true',
'es.net.ssl'='false'
'es.nodes.client.only'='false',
'es.nodes.wan.only' = 'true'
'es.net.http.auth.user'='xxxxx',
'es.net.http.auth.pass' = 'xxxxx'
);

创建成功

SELECT * FROM ctrl_rater_resumen_lla_es;

459', '*org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: 连接错误（检查网络和/或代理设置）- 所有节点都失败；试过 [[localhost:9200]] :41:6', 'org.elasticsearch.hadoop.rest.NetworkClient:execute:NetworkClient.java:152', 'org.elasticsearch.hadoop.rest.RestClient:execute:RestClient.java :424', 'org.elasticsearch.hadoop.rest.RestClient:execute:RestClient.java:388', 'org.elasticsearch.hadoop.rest.RestClient:execute:RestClient.java:392', 'org.elasticsearch.hadoop .rest.RestClient:get:RestClient.java:168', 'org.elasticsearch.hadoop.rest.RestClient:mainInfo:RestClient.java:735', 'org.elasticsearch.hadoop.rest.InitializationUtils:discoverClusterInfo:InitializationUtils.java :330'], statusCode=3), 结果=无, hasMoreRows=无) hadoop.rest.EsHadoopNoNodesLeftException：连接错误（检查网络和/或代理设置） - 所有节点都失败；试过 [[localhost:9200]] :41:6', 'org.elasticsearch.hadoop.rest.NetworkClient:execute:NetworkClient.java:152', 'org.elasticsearch.hadoop.rest.RestClient:execute:RestClient.java :424', 'org.elasticsearch.hadoop.rest.RestClient:execute:RestClient.java:388', 'org.elasticsearch.hadoop.rest.RestClient:execute:RestClient.java:392', 'org.elasticsearch.hadoop .rest.RestClient:get:RestClient.java:168', 'org.elasticsearch.hadoop.rest.RestClient:mainInfo:RestClient.java:735', 'org.elasticsearch.hadoop.rest.InitializationUtils:discoverClusterInfo:InitializationUtils.java :330'], statusCode=3), 结果=无, hasMoreRows=无) hadoop.rest.EsHadoopNoNodesLeftException：连接错误（检查网络和/或代理设置） - 所有节点都失败；试过 [[localhost:9200]] :41:6', 'org.elasticsearch.hadoop.rest.NetworkClient:execute:NetworkClient.java:152', 'org.elasticsearch.hadoop.rest.RestClient:execute:RestClient.java :424', 'org.elasticsearch.hadoop.rest.RestClient:execute:RestClient.java:388', 'org.elasticsearch.hadoop.rest.RestClient:execute:RestClient.java:392', 'org.elasticsearch.hadoop .rest.RestClient:get:RestClient.java:168', 'org.elasticsearch.hadoop.rest.RestClient:mainInfo:RestClient.java:735', 'org.elasticsearch.hadoop.rest.InitializationUtils:discoverClusterInfo:InitializationUtils.java :330'], statusCode=3), 结果=无, hasMoreRows=无) 连接错误（检查网络和/或代理设置）- 所有节点都失败；试过 [[localhost:9200]] :41:6', 'org.elasticsearch.hadoop.rest.NetworkClient:execute:NetworkClient.java:152', 'org.elasticsearch.hadoop.rest.RestClient:execute:RestClient.java :424', 'org.elasticsearch.hadoop.rest.RestClient:execute:RestClient.java:388', 'org.elasticsearch.hadoop.rest.RestClient:execute:RestClient.java:392', 'org.elasticsearch.hadoop .rest.RestClient:get:RestClient.java:168', 'org.elasticsearch.hadoop.rest.RestClient:mainInfo:RestClient.java:735', 'org.elasticsearch.hadoop.rest.InitializationUtils:discoverClusterInfo:InitializationUtils.java :330'], statusCode=3), 结果=无, hasMoreRows=无) 连接错误（检查网络和/或代理设置）- 所有节点都失败；试过 [[localhost:9200]] :41:6', 'org.elasticsearch.hadoop.rest.NetworkClient:execute:NetworkClient.java:152', 'org.elasticsearch.hadoop.rest.RestClient:execute:RestClient.java :424', 'org.elasticsearch.hadoop.rest.RestClient:execute:RestClient.java:388', 'org.elasticsearch.hadoop.rest.RestClient:execute:RestClient.java:392', 'org.elasticsearch.hadoop .rest.RestClient:get:RestClient.java:168', 'org.elasticsearch.hadoop.rest.RestClient:mainInfo:RestClient.java:735', 'org.elasticsearch.hadoop.rest.InitializationUtils:discoverClusterInfo:InitializationUtils.java :330'], statusCode=3), 结果=无, hasMoreRows=无)

标签： apache-sparkhadoopelasticsearchhivecloudera

Answer 1

The correct property is "es.nodes", not "es.node". The default is "localhost", so you are trying to connect to your localhost instead of the node you want to connect to. See the documentation for more information.

You might also want to think about whether you need the property "es.nodes.wan.only" to be set to true if you are not connecting to a cloude environment, as that will disable the automatic discovery of other nodes in your network as is explained in the documentation (you need to scroll down a bit). It will force the system to use the "es.nodes" property, which per default tries to connect to localhost. That's why you are getting the error, but even if you make it work that setting will have an impact on your performance (original emphasis):

Note that in this mode, performance is highly affected.