时间戳

首页 > 解决方案 > spark解释器读取hive metastore超时

apache-spark - spark解释器读取hive metastore超时

问题描述

大家，我正在使用 zeppelin 的 spark 解释器来读取 hive 数据。我的版本信息列表：

当我在一段spark解释器笔记本中运行这个sql时， %sql select * from the_table_not_exists 即使表不存在，它也运行300s，我认为300s是配置hive.metastore.client.socket.timeout=300

日志如下，Info 和 WARN 间隔正好 300s

INFO [2020-10-16 11:32:42,887] ({pool-2-thread-7} SchedulerFactory.java[jobStarted]:114) - Job 20200723-090353_389722279 started by scheduler org.apache.zeppelin.spark.SparkSqlInterpreter113305554
 WARN [2020-10-16 11:37:42,930] ({pool-2-thread-7} RetryingMetaStoreClient.java[invoke]:184) - MetaStoreClient lost connection. Attempting to reconnect.
org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
...
Caused by: java.net.SocketTimeoutException: Read timed out
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
    at java.net.SocketInputStream.read(SocketInputStream.java:171)
    at java.net.SocketInputStream.read(SocketInputStream.java:141)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
    at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
    ... 93 more
 INFO [2020-10-16 11:37:43,933] ({pool-2-thread-7} HiveMetaStoreClient.java[open]:376) - Trying to connect to metastore with URI thrift://host:9083
 INFO [2020-10-16 11:37:43,948] ({pool-2-thread-7} HiveMetaStoreClient.java[open]:472) - Connected to metastore.
 INFO [2020-10-16 11:37:43,956] ({pool-2-thread-7} SchedulerFactory.java[jobFinished]:120) - Job 20200723-090353_389722279 finished by scheduler org.apache.zeppelin.spark.SparkSqlInterpreter113305554

然后重新连接 hive metastore 现在我不知道如何修复它，有人可以帮助我吗？谢谢你

标签： apache-sparkhiveapache-zeppelincloudera-cdhhive-metastore

解决方案

推荐阅读