java - java.lang.NoClassDefFoundError: org/apache/spark/TaskOutputFileAlreadyExistException
问题描述
我已经阅读了 HDFS 中的数据。我分析了它,但是在写的时候我得到了这个错误。错误的继续
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/TaskOutputFileAlreadyExistException
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:167)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:104)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:102)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.doExecute(commands.scala:123)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:173)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:211)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:208)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:169)
at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:110)
at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:109)
at org.apache.spark.sql.DataFrameWriter.$anonfun$runCommand$1(DataFrameWriter.scala:828)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$4(SQLExecution.scala:100)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:160)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:87)
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:828)
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:309)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:293)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:236)
at SparkSQL.SparkHDFS.main(SparkHDFS.java:22)
我的代码
SparkSession sparkSession = SparkSession.builder().appName("FirstSQL").master("local").getOrCreate();
Encoder<MovieModal> movieModalEncoder = Encoders.bean(MovieModal.class);
Dataset<MovieModal> data = sparkSession.read().option("infershema",true)
.option("header",true)
.csv("hdfs://localhost:8020/data/ratings.csv")
.as(movieModalEncoder);
Dataset<Row> groupData = data.groupBy(new Column("movieID")).count();
groupData.write().format("csv").save("hdfs://localhost:8020/var/groupData2.csv");
解决方案
如果目录已经存在,那么我们需要在写入时提供overwrite
(覆盖现有目录)或append
(附加到目录)作为模式。
尝试:
groupData.write().mode("overwrite").format("csv").save("hdfs://localhost:8020/var/groupData2.csv");
推荐阅读
- mysql - 如何使用循环系统生成随机匹配?
- html - 带有 css houdini Paint API 的样式选择标签
- java - 在 selenium 中,当 HTML 中不存在文本但来自 api 响应时,我们可以从网页中检索文本吗?
- java - 在将 java 对象存储到 DB 中时,我们遇到了校验和失败的问题。数据库没有问题。需要及时帮助
- c# - C# 合并 tiff 文件而不增加文件大小 2-3 倍
- vue.js - 在 vuex-module-decorators 中创建可重用的 getter
- android - 活动中是否需要使用fragment?
- r - 以字符格式四舍五入数字
- javascript - Wordpress 网站上的粘性位置
- c++ - C(嵌入式):注释 FreeRTOS 的 RootTask 时代码大小不会缩小