apache-spark - Spark 2.3.0 - 引起：java.lang.ClassNotFoundException: org/apache/spark/sql/execution/WholeStageCodegenExec$$anonfun$10$$anon$1

Question

引起：java.lang.ClassNotFoundException: org/apache/spark/sql/execution/WholeStageCodegenExec$$anonfun$10$$anon$1 at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class .java:348）在 org.apache.spark.util.InnerClosureFinder$$anon$4.visitMethodInsn（ClosureCleaner.scala:484）在 org.apache.xbean.asm5.ClassReader.a（未知来源）在 org.apache.xbean .asm5.ClassReader.b(Unknown Source) at org.apache.xbean.asm5.ClassReader.accept(Unknown Source) at org.apache.xbean.asm5.ClassReader.accept(Unknown Source) at org.apache.spark.util .ClosureCleaner$.getInnerClosureClasses(ClosureCleaner.scala:87) 在 org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:224) 在 org.apache.spark。 org.apache 上的 util.ClosureCleaner$.clean(ClosureCleaner.scala:159)。spark.SparkContext.clean(SparkContext.scala:2292) at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1.apply(RDD.scala:844) at org.apache.spark.rdd.RDD$$anonfun $mapPartitionsWithIndex$1.apply(RDD.scala:843) 在 org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) 在 org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala: 112) 在 org.apache.spark.rdd.RDD.withScope(RDD.scala:363) 在 org.apache.spark.rdd.RDD.mapPartitionsWithIndex(RDD.scala:843) 在 org.apache.spark.sql.execution .WholeStageCodegenExec.doExecute(WholeStageCodegenExec.scala:608) 在 org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131) 在 org.apache.spark.sql.execution.SparkPlan $$anonfun$execute$1.apply(SparkPlan.scala:127) 在 org.apache.spark.sql.execution。SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.sql.execution.SparkPlan.executeQuery (SparkPlan.scala:152) 在 org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127) 在 org.apache.spark.sql.execution.SparkPlan.getByteArrayRdd(SparkPlan.scala:247) 在org.apache.spark.sql.execution.SparkPlan.executeCollectIterator(SparkPlan.scala:304) at org.apache.spark.sql.execution.exchange.BroadcastExchangeExec$$anonfun$relationFuture$1$$anonfun$apply$1.apply(BroadcastExchangeExec .scala:76) 在 org.apache.spark.sql.execution.exchange.BroadcastExchangeExec$$anonfun$relationFuture$1$$anonfun$apply$1.apply(BroadcastExchangeExec.scala:73) 在 org.apache.spark.sql.execution .SQL 执行$。withExecutionId(SQLExecution.scala:97) at org.apache.spark.sql.execution.exchange.BroadcastExchangeExec$$anonfun$relationFuture$1.apply(BroadcastExchangeExec.scala:72) at org.apache.spark.sql.execution.exchange。 BroadcastExchangeExec$$anonfun$relationFuture$1.apply(BroadcastExchangeExec.scala:72) at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future .scala:24) ... 还有 3 个scala:24) 在 scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) ... 还有 3 个scala:24) 在 scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) ... 还有 3 个

标签： apache-spark

Answer 1

最后我得到了答案。我在并行流中使用 spark-sql：

Tomcat升级后并行流不设置Thread.contextClassLoader

问题解决了。问题关闭