apache-spark - 实例化 JavaStreamingContext 时出现 AbstractMethodError 异常
问题描述
创建 JavaStreamingContext 时出现 AbstractMethodError 异常。我的依赖pom如下;找不到线索,谁能建议这里出了什么问题?
<dependency> <!-- Spark dependency -->
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>2.3.1</version>
<scope>provided</scope>
</dependency>
<dependency> <!-- Spark dependency -->
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.3.1</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.5</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.11</artifactId>
<version>2.2.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kafka-0-10 -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
<version>2.0.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka -->
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.10</artifactId>
<version>0.8.2.2</version>
</dependency>
org.apache.spark.streaming.scheduler.StreamingListenerBus.(StreamingListenerBus. scala:30) at org.apache.spark.streaming.scheduler.JobScheduler.(JobScheduler.scala:57) at org.apache.spark.streaming.StreamingContext.(StreamingContext.scala:184) at org.apache.spark.streaming .StreamingContext.(StreamingContext.scala:76) 在 org.apache.spark.streaming.api.java.JavaStreamingContext.(JavaStreamingContext.scala:130)
解决方案
您在这里混合了许多版本的 Spark
首先,如果您使用的是 Apache Spark 2.3.1 和 kafka 0.10+
我建议如下:
<dependency> <!-- Spark dependency -->
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>2.3.1</version>
<scope>provided</scope>
</dependency>
<dependency> <!-- Spark dependency -->
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.3.1</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.5</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.11</artifactId>
<!-- Keep the same Spark version as before -->
<version>2.3.1</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kafka-0-10 -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
<!-- Keep the same Spark version as before -->
<version>2.3.1</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka -->
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.10</artifactId>
<!-- Support for kafka version 0.8 is deprecated as of Spark 2.3.1 and you add the dependencies for kafka 0.10+ above -->
<version><your_kafka_version_0.10+></version>
</dependency>
很高兴知道您如何构建/部署您的应用程序?根据您的运行时环境,您可能需要添加一些提供的范围以防止构建的包与现有环境之间发生冲突。
希望能帮助到你
推荐阅读
- java - 为什么 Java Font 在 macOs 的 Dock 中添加 java 图标?
- mysql - MySQL 哪里不是全部为空?
- hyperledger-fabric - Hyperledger Fabric:监听数据流,同时提交交易
- reactjs - 更漂亮的冲突?(问号和点)在 typescript-react 中的 prop-bracket 中出现语法错误
- javascript - Firebase 插入发生而不是更新,没有显示错误
- google-cloud-platform - GCP 部署管理器 - 结合使用什么 Dev Ops 工具?
- django - MultipleObjectsReturned:get() 返回了多个驱动程序——它返回了 3
- ansible - 在ansible中从我的标准输出中过滤/提取一整行
- react-native - React Native 平台如何工作?
- python - 反转/解码具有多个子类别的 pandas 中的 get_dummies 编码