时间戳

首页 > 解决方案 > 运行 Spark 3.1.1 作为 Hive 3.1.2 的引擎

java - 运行 Spark 3.1.1 作为 Hive 3.1.2 的引擎

问题描述

我正在尝试使用 Spark 3.1.1（在没有 Hadoop 的情况下作为 bin 下载）作为 Hive 3.1.2 的引擎和我自己的小型 Hadoop 3.3.0 集群（1 个名称节点、1 个资源管理器、3 个数据节点），但我保留尝试对一行数据进行非常简单的插入时出现以下错误。

请注意，我在这里假设 Spark 3.1.1 与 Hive 3.1.2 兼容，这表明 spark.sql.hive.metastore.version 支持最高 3.1.2 的元存储版本。如果我错了，请纠正我。我也尝试了 Spark 2.4.8，但最终在同一个地方：

java.lang.NoClassDefFoundError: org/apache/spark/AccumulatorParam
    at org.apache.hive.spark.counter.SparkCounterGroup.createCounter(SparkCounterGroup.java:52)
    at org.apache.hive.spark.counter.SparkCounters.createCounter(SparkCounters.java:71)
    at org.apache.hive.spark.counter.SparkCounters.createCounter(SparkCounters.java:67)
    at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient$JobStatusJob.call(RemoteHiveSparkClient.java:350)
    at org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:378)
    at org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:343)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.AccumulatorParam
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
    ... 10 more

2021-06-14T20:30:41,857  INFO [HiveServer2-Background-Pool: Thread-42] reexec.ReOptimizePlugin: ReOptimization: retryPossible: false
2021-06-14T20:30:41,857 ERROR [HiveServer2-Background-Pool: Thread-42] ql.Driver: FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Spark job failed during runtime. Please check stacktrace for the root cause.
2021-06-14T20:30:41,857  INFO [HiveServer2-Background-Pool: Thread-42] ql.Driver: Completed executing command(queryId=hive_20210614203032_efd9edb9-cd18-40ef-80c1-be1d1d30dde4); Time taken: 7.494 seconds
2021-06-14T20:30:41,858  INFO [HiveServer2-Background-Pool: Thread-42] ql.Driver: Concurrency mode is disabled, not creating a lock manager
2021-06-14T20:30:41,874 ERROR [HiveServer2-Background-Pool: Thread-42] operation.Operation: Error running hive query:
org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Spark job failed during runtime. Please check stacktrace for the root cause.
    at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:335) ~[hive-service-3.1.2.jar:3.1.2]
    at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:226) ~[hive-service-3.1.2.jar:3.1.2]
    at org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87) ~[hive-service-3.1.2.jar:3.1.2]
    at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:316) ~[hive-service-3.1.2.jar:3.1.2]
    at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_292]
    at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_292]
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845) ~[hadoop-common-3.3.0.jar:?]
    at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:329) ~[hive-service-3.1.2.jar:3.1.2]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_292]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_292]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_292]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_292]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Spark job failed during runtime. Please check stacktrace for the root cause.
    at org.apache.hadoop.hive.ql.exec.spark.SparkTask.getSparkJobInfo(SparkTask.java:498) ~[hive-exec-3.1.2.jar:3.1.2]
    at org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:156) ~[hive-exec-3.1.2.jar:3.1.2]
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205) ~[hive-exec-3.1.2.jar:3.1.2]
    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97) ~[hive-exec-3.1.2.jar:3.1.2]
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2664) ~[hive-exec-3.1.2.jar:3.1.2]
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2335) ~[hive-exec-3.1.2.jar:3.1.2]
    at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2011) ~[hive-exec-3.1.2.jar:3.1.2]
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1709) ~[hive-exec-3.1.2.jar:3.1.2]
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1703) ~[hive-exec-3.1.2.jar:3.1.2]
    at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157) ~[hive-exec-3.1.2.jar:3.1.2]
    at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:224) ~[hive-service-3.1.2.jar:3.1.2]
    ... 11 more
Caused by: org.apache.spark.SparkException: java.lang.NoClassDefFoundError: org/apache/spark/AccumulatorParam
    at org.apache.hive.spark.counter.SparkCounterGroup.createCounter(SparkCounterGroup.java:52)
    at org.apache.hive.spark.counter.SparkCounters.createCounter(SparkCounters.java:71)
    at org.apache.hive.spark.counter.SparkCounters.createCounter(SparkCounters.java:67)
    at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient$JobStatusJob.call(RemoteHiveSparkClient.java:350)
    at org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:378)
    at org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:343)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.AccumulatorParam
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
    ... 10 more

    at org.apache.hive.spark.client.SparkClientImpl$ClientProtocol.handle(SparkClientImpl.java:595) ~[hive-exec-3.1.2.jar:3.1.2]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_292]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_292]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_292]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_292]
    at org.apache.hive.spark.client.rpc.RpcDispatcher.handleCall(RpcDispatcher.java:121) ~[hive-exec-3.1.2.jar:3.1.2]
    at org.apache.hive.spark.client.rpc.RpcDispatcher.channelRead0(RpcDispatcher.java:80) ~[hive-exec-3.1.2.jar:3.1.2]
    at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
    at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
    at io.netty.handler.codec.ByteToMessageCodec.channelRead(ByteToMessageCodec.java:103) ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:138) ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
    ... 1 more

但是，JAR scala-library、spark-core、spark-unsafe 和 spark-network-common 从 Spark (/opt/spark/jars) 链接到 Hive 库 (/opt/hive/lib)

我的 hive-site.xml 看起来像这样

<configuration>
<property>
  <name>hive.metastore.transactional.event.listeners</name>
  <value>
    org.apache.hive.hcatalog.listener.DbNotificationListener,
    org.apache.kudu.hive.metastore.KuduMetastorePlugin
  </value>
</property>
<property>
  <name>hive.metastore.disallow.incompatible.col.type.changes</name>
  <value>false</value>
</property>
<property>
  <name>hive.metastore.notifications.add.thrift.objects</name>
  <value>true</value>
</property>
<property>
  <name>hive.metastore.uris</name>
  <value>thrift://Alan.local:9083,thrift://Carl.local:9083</value>
</property>
<property>
   <name>hive.metastore.port</name>
   <value>9083</value>
   <description>Hive metastore listener port</description>
</property>
<property>
   <name>hive.metastore.warehouse.dir</name>
   <value>/user/hive/warehouse</value>
   <description>Hive metastore listener port</description>
</property>
<property>
   <name>javax.jdo.option.ConnectionURL</name>
   <value>jdbc:mariadb://Alan.local/metastore?allowPublicKeyRetrieval=true</value>
</property>
<property>
  <name>javax.jdo.option.ConnectionDriverName</name>
  <value>org.mariadb.jdbc.Driver</value>
</property>

<property>
  <name>javax.jdo.option.ConnectionUserName</name>
  <value>hive</value>
</property>

<property>
  <name>javax.jdo.option.ConnectionPassword</name>
  <value>PASSWORD</value>
</property>

<property>
  <name>datanucleus.autoCreateSchema</name>
  <value>false</value>
</property>

<property>
  <name>datanucleus.fixedDatastore</name>
  <value>true</value>
</property>

<property>
  <name>datanucleus.autoStartMechanism</name>
  <value>SchemaTable</value>
</property>
<property>
  <name>hive.metastore.event.db.notification.api.auth</name>
  <value>false</value>
</property>
<property>
 <name>hive.cluster.delegation.token.store.class</name>
 <value>org.apache.hadoop.hive.thrift.DBTokenStore</value>
</property>
<property>
 <name>hive.server2.authentication</name>
 <value>NOSASL</value>
</property>
<property>
 <name>hive.server2.enable.doAs</name>
 <value>false</value>
</property>
<property>
 <name>hive.metastore.execute.setugi</name>
 <value>true</value>
</property>
<property>
 <name>hive.execution.engine</name>
 <value>spark</value>
</property>
<property>
 <name>spark.master</name>
 <value>spark://Alan.local:7077</value>
</property>
<property>
 <name>spark.eventLog.enabled</name>
 <value>true</value>
</property>
<property>
 <name>spark.eventLog.dir</name>
 <value>/opt/spark/logs</value>
</property>
<property>
 <name>spark.executor.memory</name>
 <value>512m</value>
</property>
<property>
 <name>spark.serializer</name>
 <value>org.apache.spark.serializer.KryoSerializer</value>
</property>
<property>
 <name>spark.sql.hive.metastore.version</name>
 <value>3.1.2</value>
</property>
<property>
 <name>spark.sql.warehouse.dir</name>
 <value>/user/hive/warehouse</value>
</property>
<!-- <property>
 <name>spark.yarn.jars</name>
 <value>hdfs://Alan.local:8020/lib/spark</value>
</property> -->
<property>
    <name>hive.server2.active.passive.ha.enable</name>
    <value>true</value>
</property>
</configuration>

spark-defaults.conf 是这样的：

spark.master                     spark://Alan.local:7077
spark.eventLog.enabled           true
spark.eventLog.dir               /opt/spark/logs/events
#spark.eventLog.dir               hdfs://namenode:8021/directory
spark.serializer                 org.apache.spark.serializer.KryoSerializer
spark.driver.memory              512m
#spark.executor.extraJavaOptions  -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"

我正在运行一个 Spark Master 和一个 Spark Worker（Slave）。查看 master 的 Web UI，我可以看到它会显示为 /opt/spark/jars/* 在加载的路径中：

Spark Executor Command: "/usr/lib/jvm/java-8-openjdk-amd64/bin/java" "-cp" "/opt/spark/conf/:/opt/spark/jars/*:/opt/hadoop/etc/hadoop/:/opt/hadoop/share/hadoop/common/lib/*:/opt/hadoop/share/hadoop/common/*:/opt/hadoop/share/hadoop/hdfs/:/opt/hadoop/share/hadoop/hdfs/lib/*:/opt/hadoop/share/hadoop/hdfs/*:/opt/hadoop/share/hadoop/mapreduce/*:/opt/hadoop/share/hadoop/yarn/:/opt/hadoop/share/hadoop/yarn/lib/*:/opt/hadoop/share/hadoop/yarn/*" "-Xmx512M" "-Dspark.hadoop.hbase.master.port=16000" "-Dspark.hadoop.hbase.regionserver.port=16020" "-Dspark.driver.port=41415" "-Dspark.hadoop.hbase.rest.port=8080" "-Dspark.hadoop.hbase.status.multicast.address.port=16100" "-Dspark.rpc.askTimeout=10s" "-Dspark.hadoop.hbase.regionserver.info.port=16030" "-Dspark.hadoop.hbase.master.info.port=16010" "-Dhive.spark.log.dir=/opt/spark/logs/" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://CoarseGrainedScheduler@Carl.local:41415" "--executor-id" "0" "--hostname" "10.0.0.32" "--cores" "1" "--app-id" "app-20210614203039-0000" "--worker-url" "spark://Worker@10.0.0.32:45725"
========================================

为了让我觉得更奇怪，/opt/spark-2.4.8/jars/spark-core_2.12-2.4.8.jar | grep AccumulatorParam 返回 AccumulatorParam.class，但 3.1.1 没有。然而，相同的设置（mv 重命名文件夹、重新链接 jar、将配置文件复制到正确的位置）在 Spark 2.4.8 和 3.1.1 中都会出现相同的错误。我在 3.1.2 的修复列表中没有看到任何相关内容，所以我没有尝试过那个版本。

所以总结和重申：

如果有人能推动我前进，我将不胜感激。到目前为止，使用 Hive 设置 Spark 一直是最大的麻烦（在我的集群中，我在 Hive 上运行了 Impala，它以最小的努力完美地工作，而使用 Spark，我花了数小时处理通常由 StackOverflow 和其他站点解决的不同错误，但有了这个，我只需要放弃并寻求一些方向）。

标签： javaapache-sparkhive

解决方案

推荐阅读