hadoop - 找不到 org.apache.hadoop.conf.Configuration
问题描述
我有一个 java-maven 项目。它将文件从本地复制到 HDFS。我的 pom.xml 中有 hadoop-common。尽管如此,它仍然失败并出现以下错误:
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
at apache.spark.poc.utils.FileProcessor.<clinit>(FileProcessor.java:60)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 1 more
以下是我的 pom.xml 中的依赖项:这里有什么我遗漏的吗?我还通过 IDE 检查了它,我可以看到 jar 下的 hadoop-common 和 Configuration 类。
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.2.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.1.1</version>
<scope>provided</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming_2.10 -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.11</artifactId>
<version>2.1.1</version>
<scope>provided</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kafka_2.10 -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kafka_2.10</artifactId>
<version>1.6.3</version>
<scope>provided</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql_2.10 -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>2.1.1</version>
<scope>provided</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql-kafka-0-10_2.11 -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql-kafka-0-10_2.11</artifactId>
<version>2.1.1</version>
<!-- <scope>provided</scope> -->
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients -->
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>0.10.2.1</version>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.11</artifactId>
<version>0.11.0.0</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.6.3</version>
</dependency>
<!-- https://mvnrepository.com/artifact/mysql/mysql-connector-java -->
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.44</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.commons/commons-dbcp2 -->
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-dbcp2</artifactId>
<version>2.0</version>
</dependency>
<dependency>
<groupId>commons-lang</groupId>
<artifactId>commons-lang</artifactId>
<version>2.6</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.1</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common -->
解决方案
发现问题。它的范围与提供的一样。我对这些库进行了一项一项分析,并检查了所有正在部署的内容。
hadoop-common 库与 spark-core 一起被选中。但是,spark-core 被标记为提供范围。因此,它在编译阶段可用,但在我通过 IntelliJ 运行时不可用。
问题已修复。
推荐阅读
- python-3.x - 无法将“#N/A”读取为字符串
- angular - SignalR .Net Core 多连接
- kotlin - 如何检测用户是否按下后退按钮并在 Kotlin 片段内的应用程序中导航?
- wordpress - 在指向我的域的 Ubuntu 18.04 VPS 上部署 Kubernetes Cluster + Wordpress
- r - R中的季节性温度箱线图
- r - 在 Shiny 的地图上绘制散点图
- php - PHP 联系页面有效但显示值代码
- c# - 我的脚本使用光线投射随着时间的推移造成伤害,导致 Unity 崩溃。有谁知道为什么?
- reactjs - 在 useState 挂钩中将一个状态与另一个状态相关联
- socat - socat virt tty 硬件流控