apache-spark - 将 spark 版本升级到 2.4.3：java.lang.ClassNotFoundException：org.apache.hadoop.fs.StorageStatistics

Question

我已将 spark-version 升级到 2.4.3，但我发现了这个异常

我错过了什么吗？

    To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
    java.lang.NoClassDefFoundError: org/apache/hadoop/fs/StorageStatistics
      at java.lang.Class.forName0(Native Method)
      at java.lang.Class.forName(Class.java:348)
      at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2134)
      at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2099)
      at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)
      at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2654)
      at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667)
      at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
      at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
      at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
      at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
      at org.apache.spark.util.Utils$.getHadoopFileSystem(Utils.scala:1866)
      at org.apache.spark.scheduler.EventLoggingListener.<init>(EventLoggingListener.scala:71)
      at org.apache.spark.SparkContext.<init>(SparkContext.scala:521)
      at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
      at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
      at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926)
      at scala.Option.getOrElse(Option.scala:121)
      at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
      at org.apache.spark.repl.Main$.createSparkSession(Main.scala:106)
      ... 62 elided
    Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.StorageStatistics
      at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
      at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
      ... 82 more
    <console>:14: error: not found: value spark
           import spark.implicits._
                  ^
    <console>:14: error: not found: value spark
           import spark.sql
    These are the versions for aws sdk and hadoop-aws
    I have upgraded the spark-version to 2.4.3, but I found this exception 

我错过了什么吗？

我已配置的 aws sdk jar

hadoop-aws-3.2.0.jar
aws-java-sdk-bundle-1.11.619.jar

标签： apache-spark