apache-spark - Spark 错误：`线程“主”java.lang.NoClassDefFoundError 中的异常：org/apache/spark/sql/execution/streaming/Source$class`

Question

更新：问题已修复。是版本问题。需要将 spark-core 等更改为 2.2.0 版。

我是 Apache Spark 的新手。我正在尝试通过 Kafka 发送日志并使用 Spark Streaming 处理它们，然后再将其传递给 ElasticSearch。但是我收到以下错误。

org.apache.spark.sql.kafka010.KafkaSource.(KafkaSource.scala:87) at org.apache.spark.sql.kafka010.KafkaSource.(KafkaSource.scala:87) at org.apache.spark.sql.kafka010.KafkaSource. apache.spark.sql.kafka010.KafkaSourceProvider.createSource(KafkaSourceProvider.scala:152) 在 org.apache.spark.sql.execution.datasources.DataSource.createSource(DataSource.scala:240) 在 org.apache.spark.sql。 streaming.StreamingQueryManager$$anonfun$1.applyOrElse(StreamingQueryManager.scala:245) 在 org.apache.spark.sql.streaming.StreamingQueryManager$$anonfun$1.applyOrElse(StreamingQueryManager.scala:241) 在 org.apache.spark.sql。 org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:27​​9) 上的催化剂.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:27​​9)。 spark.sql.catalyst.trees。CurrentOrigin$.withOrigin(TreeNode.scala:69) at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:27​​8) at org.apache.spark.sql.catalyst.trees.TreeNode.transform (TreeNode.scala:268) 在 org.apache.spark.sql.streaming.StreamingQueryManager.startQuery(StreamingQueryManager.scala:241) 在 org.apache.spark.sql.streaming.DataStreamWriter.start(DataStreamWriter.scala:287) 在com.fork.SreamingApp.App.main(App.java:78) 引起：java.lang.ClassNotFoundException: org.apache.spark.sql.execution.streaming.Source$class at java.base/jdk.internal.loader .BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582) 在 java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:190) 在 java.base/java.lang.ClassLoader.loadClass(ClassLoader.java :499) ... 13 更多scala:69) 在 org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:27​​8) 在 org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:268)在 org.apache.spark.sql.streaming.StreamingQueryManager.startQuery(StreamingQueryManager.scala:241) 在 org.apache.spark.sql.streaming.DataStreamWriter.start(DataStreamWriter.scala:287) 在 com.fork.SreamingApp.App .main(App.java:78) 引起：java.lang.ClassNotFoundException: org.apache.spark.sql.execution.streaming.Source$class at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader. java:582) 在 java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:190) 在 java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:499) ... 13更多的scala:69) 在 org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:27​​8) 在 org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:268)在 org.apache.spark.sql.streaming.StreamingQueryManager.startQuery(StreamingQueryManager.scala:241) 在 org.apache.spark.sql.streaming.DataStreamWriter.start(DataStreamWriter.scala:287) 在 com.fork.SreamingApp.App .main(App.java:78) 引起：java.lang.ClassNotFoundException: org.apache.spark.sql.execution.streaming.Source$class at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader. java:582) 在 java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:190) 在 java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:499) ... 13更多的org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:268) 的 transformDown(TreeNode.scala:27​​8) org.apache.spark.sql.streaming.StreamingQueryManager.startQuery(StreamingQueryManager.scala: 241) at org.apache.spark.sql.streaming.DataStreamWriter.start(DataStreamWriter.scala:287) at com.fork.SreamingApp.App.main(App.java:78) 引起：java.lang.ClassNotFoundException: org .apache.spark.sql.execution.streaming.Source$class 在 java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582) 在 java.base/jdk.internal.loader.ClassLoaders$AppClassLoader。 loadClass(ClassLoaders.java:190) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:499) ... 还有 13 个org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:268) 的 transformDown(TreeNode.scala:27​​8) org.apache.spark.sql.streaming.StreamingQueryManager.startQuery(StreamingQueryManager.scala: 241) at org.apache.spark.sql.streaming.DataStreamWriter.start(DataStreamWriter.scala:287) at com.fork.SreamingApp.App.main(App.java:78) 引起：java.lang.ClassNotFoundException: org .apache.spark.sql.execution.streaming.Source$class 在 java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582) 在 java.base/jdk.internal.loader.ClassLoaders$AppClassLoader。 loadClass(ClassLoaders.java:190) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:499) ... 还有 13 个268) 在 org.apache.spark.sql.streaming.StreamingQueryManager.startQuery(StreamingQueryManager.scala:241) 在 org.apache.spark.sql.streaming.DataStreamWriter.start(DataStreamWriter.scala:287) 在 com.fork.SreamingApp .App.main(App.java:78) 引起：java.lang.ClassNotFoundException: org.apache.spark.sql.execution.streaming.Source$class at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass( BuiltinClassLoader.java:582) 在 java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:190) 在 java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:499) .. . 13 更多268) 在 org.apache.spark.sql.streaming.StreamingQueryManager.startQuery(StreamingQueryManager.scala:241) 在 org.apache.spark.sql.streaming.DataStreamWriter.start(DataStreamWriter.scala:287) 在 com.fork.SreamingApp .App.main(App.java:78) 引起：java.lang.ClassNotFoundException: org.apache.spark.sql.execution.streaming.Source$class at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass( BuiltinClassLoader.java:582) 在 java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:190) 在 java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:499) .. . 13 更多78) 引起：java.lang.ClassNotFoundException: org.apache.spark.sql.execution.streaming.Source$class at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582) at java。 base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:190) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:499) ... 13 更多78) 引起：java.lang.ClassNotFoundException: org.apache.spark.sql.execution.streaming.Source$class at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582) at java。 base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:190) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:499) ... 13 更多

这是我的代码文件

应用程序.java

public static void main(String[] args) throws NotSerializableException, InterruptedException {
        App streamingApp = new App();
        SparkSession spark = SparkSession
                  .builder()
                  .appName("Java Spark SQL data sources example")
                  .master("local")
                  .getOrCreate();

        Dataset<Row> ds1 = spark
                  .readStream()
                  .format("kafka")
                  .option("kafka.bootstrap.servers", "localhost:32770")
                  .option("subscribe", "context3")
                  .load();

        StreamingQuery query = ds1.writeStream()
                  .format("console")
                  .start();

        query.awaitTermination();
        System.out.println("DONE");



}

pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0" 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

 <groupId>com.fork</groupId>
 <artifactId>SreamingApp</artifactId>
 <version>0.0.1-SNAPSHOT</version>
 <packaging>jar</packaging>

 <name>SreamingApp</name>
 <url>http://maven.apache.org</url>

 <properties>
   <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
 </properties>

 <dependencies>
   <dependency>
     <groupId>com.maxmind.geoip2</groupId>
     <artifactId>geoip2</artifactId>
     <version>2.12.0</version>
   </dependency>


   <dependency>
     <groupId>junit</groupId>
     <artifactId>junit</artifactId>
     <version>3.8.1</version>
     <scope>test</scope>
   </dependency>

   <dependency>
     <groupId>org.apache.spark</groupId>
     <artifactId>spark-core_2.11</artifactId>
     <version>2.0.0</version>
   </dependency>

   <dependency>
     <groupId>org.mongodb.spark</groupId>
     <artifactId>mongo-spark-connector_2.11</artifactId>
     <version>2.2.3</version>
   </dependency>

    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.11</artifactId>
        <version>2.0.0</version>
    </dependency>

    <dependency>
        <groupId>com.google.code.gson</groupId>
        <artifactId>gson</artifactId>
        <version>2.8.5</version>
    </dependency>


    <dependency>
        <groupId>org.mongodb</groupId>
        <artifactId>bson</artifactId>
        <version>3.8.0</version>
    </dependency>


    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming_2.11</artifactId>
        <version>2.0.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql-kafka-0-10_2.11</artifactId>
        <version>2.0.2</version>
    </dependency>


    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
        <version>2.0.0</version>
    </dependency>

    <!-- <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-streaming-kafka_2.11</artifactId> 
        <version>1.6.3</version> </dependency> -->

    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka-clients</artifactId>
        <version>1.1.0</version>
    </dependency>




    <dependency>
        <groupId>com.fasterxml.jackson.module</groupId>
        <artifactId>jackson-module-scala_2.11</artifactId>
        <version>2.9.5</version>
    </dependency>

</dependencies>

我该如何解决？请帮帮我。

标签： apache-sparkapache-kafkaspark-structured-streaming