时间戳

首页 > 解决方案 > 无法在独立模式下运行 Spark 程序（客户端和集群模式下的错误）

scala - 无法在独立模式下运行 Spark 程序（客户端和集群模式下的错误）

问题描述

我有一个单独的 Ubuntu 服务器，我在其中运行了一个主服务器和一个从属服务器（一个执行程序），它们显示在 8080 UI 上。
我可以spark-shell --master spark://foo.bar:7077成功运行，但我无法成功提交我的程序（fat jar）并且出现错误（独立模式）。

我有一个Main对象，它extends App没有一个 main 方法。并且全部在一个package myProject. 我正在像这样运行我的程序：

 spark-submit --master spark://foo.bar:7077 \
--class myProject.Main \
--deploy-mode client \
--num-executors 1 \
--executor-memory 58g \
--executor-cores 39 \
--driver-memory 4g \
--driver-cores 2 \
--conf spark.driver.memoryOverhead=819m \
--conf spark.executor.memoryOverhead=819m \
target/scala-2.12/myProject-assembly-0.1.jar

client模式的输出：

Exception in thread "main" java.lang.NoSuchMethodError: scala.App.$init$(Lscala/App;)V
        at mstproject.Main$.<init>(Main.scala:8)
        at mstproject.Main$.<clinit>(Main.scala)
        at mstproject.Main.main(Main.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:855)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:930)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:939)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
log4j:WARN No appenders could be found for logger (org.apache.spark.util.ShutdownHookManager).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

我已经检查了一个类似的错误，但是我的所有软件包似乎都与Scala 2.12兼容，因为我的build.sbt显示（我不确定我的assemblyMergeStrategy）：

scalaVersion := "2.12.12"
libraryDependencies ++= Seq(
  "com.github.pathikrit" %% "better-files" % "3.9.1",
  "org.scalatest" %% "scalatest" % "3.2.3" % Test,
  "org.apache.spark" %% "spark-core" % "2.4.8",
  "org.apache.spark" %% "spark-sql" % "2.4.8",
  "org.apache.spark" %% "spark-graphx" % "2.4.8",
  "redis.clients" % "jedis" % "3.5.1",
  "com.redislabs" %% "spark-redis" % "2.4.2"
)

assemblyMergeStrategy in assembly := {
  case PathList("org","aopalliance", xs @ _*) => MergeStrategy.last
  case PathList("javax", "inject", xs @ _*) => MergeStrategy.last
  case PathList("javax", "servlet", xs @ _*) => MergeStrategy.last
  case PathList("javax", "activation", xs @ _*) => MergeStrategy.last
  case PathList("org", "apache", xs @ _*) => MergeStrategy.last
  case PathList("com", "google", xs @ _*) => MergeStrategy.last
  case PathList("com", "esotericsoftware", xs @ _*) => MergeStrategy.last
  case PathList("com", "codahale", xs @ _*) => MergeStrategy.last
  case PathList("com", "yammer", xs @ _*) => MergeStrategy.last
  case "about.html" => MergeStrategy.rename
  case "META-INF/ECLIPSEF.RSA" => MergeStrategy.last
  case "META-INF/mailcap" => MergeStrategy.last
  case "META-INF/mimetypes.default" => MergeStrategy.last
  case PathList("META-INF", xs @ _*) =>
    xs map {_.toLowerCase} match {
      case "manifest.mf" :: Nil | "index.list" :: Nil | "dependencies" :: Nil =>
        MergeStrategy.discard
      case ps @ x :: xs if ps.last.endsWith(".sf") || ps.last.endsWith(".dsa") =>
        MergeStrategy.discard
      case "plexus" :: xs =>
        MergeStrategy.discard
      case "services" :: xs =>
        MergeStrategy.filterDistinctLines
      case "spring.schemas" :: Nil | "spring.handlers" :: Nil =>
        MergeStrategy.filterDistinctLines
      case _ => MergeStrategy.first
    }
  case "application.conf" => MergeStrategy.concat
  case "reference.conf" => MergeStrategy.concat
  case "plugin.properties" => MergeStrategy.last
  case "log4j.properties" => MergeStrategy.last
  case _ => MergeStrategy.first
//  case x =>
//    val oldStrategy = (assemblyMergeStrategy in assembly).value
//    oldStrategy(x)
}

cluster模式的输出：

log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
21/07/17 17:35:40 INFO SecurityManager: Changing view acls to: root
21/07/17 17:35:40 INFO SecurityManager: Changing modify acls to: root
21/07/17 17:35:40 INFO SecurityManager: Changing view acls groups to:
21/07/17 17:35:40 INFO SecurityManager: Changing modify acls groups to:
21/07/17 17:35:40 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
21/07/17 17:35:40 INFO Utils: Successfully started service 'driverClient' on port 34218.
21/07/17 17:35:40 INFO TransportClientFactory: Successfully created connection to foo.bar/10.0.8.137:7077 after 39 ms (0 ms spent in bootstraps)
21/07/17 17:35:41 INFO ClientEndpoint: Driver successfully submitted as driver-20210717173541-0003
21/07/17 17:35:41 INFO ClientEndpoint: ... waiting before polling master for driver state
21/07/17 17:35:46 INFO ClientEndpoint: ... polling master for driver state
21/07/17 17:35:46 INFO ClientEndpoint: State of driver-20210717173541-0003 is FAILED
21/07/17 17:35:46 INFO ShutdownHookManager: Shutdown hook called
21/07/17 17:35:46 INFO ShutdownHookManager: Deleting directory /tmp/spark-c15b4457-664f-43b7-9699-b62839ec83c0

21/07/17 17:45:08 INFO Master: Driver submitted org.apache.spark.deploy.worker.DriverWrapper
21/07/17 17:45:08 INFO Master: Launching driver driver-20210717174508-0007 on worker worker-20210716205255-10.0.8.137-45558
21/07/17 17:45:12 INFO Master: Removing driver: driver-20210717174508-0007
21/07/17 17:45:14 INFO Master: 10.0.8.137:34806 got disassociated, removing it.
21/07/17 17:45:14 INFO Master: 10.0.8.137:38394 got disassociated, removing it.

deploy-mode我觉得两个s都发生了同样的事情。client通过 spark-submit 可以看到错误，但是模式中的错误是cluster隐藏的，必须从 Master UI 中检查（我不明白为什么它在 Master 和 Worker 日志文件中不可见。

更新：正如 Luis 所说，这是 Scala 不兼容的问题，我的 Spark 集群使用的是嵌入式 Scala 2.11 而不是 2.12。所以通过将我的 fat jar Scala 版本降级到 2.11 来修复它。

标签： scalaapache-sparkubuntu-serverapache-spark-standalone

解决方案

推荐阅读