apache-spark - Apache Spark:SparkFiles.get(fileName.txt) - 无法从 SparkContext 检索文件内容
问题描述
我使用SparkContext.addFile("hdfs://host:54310/spark/fileName.txt")
并添加了一个文件到SparkContext
. 我使用org.apache.spark.SparkFiles.get(fileName.txt)
. 它显示了一条绝对路径,类似于/tmp/spark-xxxx/userFiles-xxxx/fileName.txt
.
现在我想从上面给定的绝对路径位置读取该文件
SparkContext
。我试过sc.textFile(org.apache.spark.SparkFiles.get("fileName.txt")).collect().foreach(println)
它认为返回的路径SparkFiles.get()
为HDFS 路径,这是不正确的。
我进行了广泛搜索以找到任何有用的阅读材料,但运气不佳。
方法有什么问题吗?非常感谢任何帮助。
这是代码和结果:
scala> sc.addFile("hdfs://localhost:54310/spark/fileName.txt")
scala> org.apache.spark.SparkFiles.get("fileName.txt")
res23: String = /tmp/spark-3646b5fe-0a67-4a16-bd25-015cc73533cd/userFiles-a7d54640-fab2-4dfa-a94f-7de6f74a0764/fileName.txt
scala> sc.textFile(org.apache.spark.SparkFiles.get("fileName.txt")).collect().foreach(println)
org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://localhost:54310/tmp/spark-3646b5fe-0a67-4a16-bd25-015cc73533cd/userFiles-a7d54640-fab2-4dfa-a94f-7de6f74a0764/fileName.txt
at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:287)
at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:229)
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:315)
at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:200)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2092)
at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:939)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
at org.apache.spark.rdd.RDD.collect(RDD.scala:938)
... 49 elided
解决方案
使用“file://”语法引用本地文件。
sc.textFile("file://" + org.apache.spark.SparkFiles.get("fileName.txt"))
.collect()
.foreach(println)
推荐阅读
- javascript - 单击按钮以选择输入中的文本不起作用
- laravel - Laravel 图像不显示
- php - Timber posts_per_page 仅在存档中显示一篇帖子
- biztalk - 是否可以让 2 个 BizTalk Server 节点从单个 MessageBox 运行?
- c# - DateTime 作为类通用约束
- flutter - 在列表生成器中使用单选按钮
- javascript - 同步函数的执行速度比异步函数快吗?
- cassandra - 为什么基于第一个聚类键的查询需要 ALLOW FILTERING?
- elasticsearch - Elasticsearch path.logs 无法正常工作
- java - 继承时获取泛型参数的泛型参数