python - 阶段失败时的 Spark FileAlreadyExistsException
问题描述
我正在尝试在重新分区后将数据帧写入 s3 位置。但是每当写入阶段失败并且 Spark 重试该阶段时,它就会抛出 FileAlreadyExistsException。
当我重新提交作业时,如果 spark 一次尝试完成该阶段,它工作正常。
下面是我的代码块
df.repartition(<some-value>).write.format("orc").option("compression", "zlib").mode("Overwrite").save(path)
我相信 Spark 应该在重试之前从失败的阶段删除文件。我知道如果我们将重试设置为零,这将得到解决,但火花阶段预计会失败,这不是一个合适的解决方案。
以下是错误:
Job aborted due to stage failure: Task 0 in stage 6.1 failed 4 times, most recent failure: Lost task 0.3 in stage 6.1 (TID 740, ip-address, executor 170): org.apache.hadoop.fs.FileAlreadyExistsException: File already exists:s3://<bucket-name>/<path-to-object>/part-00000-c3c40a57-7a50-41da-9ce2-555753cab63a-c000.zlib.orc
at com.amazon.ws.emr.hadoop.fs.s3.upload.plan.RegularUploadPlanner.checkExistenceIfNotOverwriting(RegularUploadPlanner.java:36)
at com.amazon.ws.emr.hadoop.fs.s3.upload.plan.RegularUploadPlanner.plan(RegularUploadPlanner.java:30)
at com.amazon.ws.emr.hadoop.fs.s3.upload.plan.UploadPlannerChain.plan(UploadPlannerChain.java:37)
at com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem.create(S3NativeFileSystem.java:601)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:932)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:913)
at com.amazon.ws.emr.hadoop.fs.EmrFileSystem.create(EmrFileSystem.java:242)
at org.apache.orc.impl.PhysicalFsWriter.<init>(PhysicalFsWriter.java:95)
at org.apache.orc.impl.WriterImpl.<init>(WriterImpl.java:170)
at org.apache.orc.OrcFile.createWriter(OrcFile.java:843)
at org.apache.orc.mapreduce.OrcOutputFormat.getRecordWriter(OrcOutputFormat.java:50)
at org.apache.spark.sql.execution.datasources.orc.OrcOutputWriter.<init>(OrcOutputWriter.scala:43)
at org.apache.spark.sql.execution.datasources.orc.OrcFileFormat$$anon$1.newInstance(OrcFileFormat.scala:121)
at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:120)
at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.<init>(FileFormatDataWriter.scala:108)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:233)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:169)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:168)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:121)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Driver stacktrace:
我正在使用带有 EMR 的 Spark 2.4,请提出解决方案。
编辑1:请注意这个问题与覆盖模式无关,我已经在使用它了。正如问题标题所暗示的那样,问题在于阶段失败时的剩余文件。可能是 Spark UI 清除了它。
解决方案
spark.hadoop.orc.overwrite.output.file=true
在您的 Spark 配置中设置。
您可以在此处找到有关此配置的更多详细信息 - OrcConf.java
推荐阅读
- python - index 必须是单调递增或递减的
- reactjs - 显示 React Skeleton Loader,直到图像完全加载
- sql-server - 如何从系统版本表中获取多个日期
- vba - 如何逐行修改文件
- ios - 拖放过滤原理
- java - 如何从 xml 集成创建 bean
- spring - 角色的 Spring Spel
- css - 为什么 CSS 在 android 上无法正常工作?
- spring - 如何将此“Group By”MySQL 查询转换为 Spring 的 JPA / CrudRepository 接口
- java - 我在 Android 4.4 中运行时构建的 jar 中的“java.lang.UnsatisfiedLinkError”。*