时间戳

首页 > 解决方案 > 写入 aws s3 存储桶时 Spark 作业失败 -

amazon-web-services - 写入 aws s3 存储桶时 Spark 作业失败 -

问题描述

写入 aws s3 存储桶时 Spark 作业失败，我收到 java.io.FileNotFoundException: No such file or directory

堆栈跟踪：

java.io.FileNotFoundException: No such file or directory: s3a://vishal/test/abc.parquet/_temporary/0/task_20190422091705_0001_m_000000 
at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:993) 
at org.apache.hadoop.fs.s3a.S3AFileSystem.listStatus(S3AFileSystem.java:734) 
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.mergePaths(FileOutputCommitter.java:360) 
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:310) 
at org.apache.parquet.hadoop.ParquetOutputCommitter.commitJob(ParquetOutputCommitter.java:48) 
at org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.commitJob(HadoopMapReduceCommitProtocol.scala:166) 
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:185) 
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:159) 
at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:104) 
at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:102) 
at org.apache.spark.sql.execution.command.DataWritingCommandExec.doExecute(commands.scala:122) 
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131) 
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127) 
at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155) 
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) 
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152) 
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127) 
at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80) 
at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80) 
at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:668) 
at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:668) 
at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78) 
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125) 
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73) 
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:668) 
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:276) 
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:270) 
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:228) 
at org.apache.spark.sql.DataFrameWriter.parquet(DataFrameWriter.scala:557)

标签： amazon-web-servicesapache-sparkamazon-s3

解决方案

很可能是权限问题。从您的异常来看，Spark 作业似乎缺少 AWS S3 对GetObject. 确保您的 Spark 作业至少具有读取、写入和移动权限。

但是，如果您使用 Spark 写入 S3 位置并期望该文件立即可用于 Spark，那么它不会每次都有效，因为

S3 可能需要一些时间才能使新创建的对象出现在目录列表中，而删除的对象可能仍然可见。

如果您仍然需要这样做，我建议您在文件最终移动到 S3 后，在可靠的持久存储（如本地（即磁盘或 HDFS））上执行所有与文件相关的操作。

但是，如果您没有任何其他选择，请尝试S3Guard。

推荐阅读