apache-spark - Spark - 从 hdfs 读取隐藏文件

Question

我正在使用 pyspark shell 来分析 hdfs 中的数据。hdfs 路径中有隐藏文件，我想通过 shell 读取它们。然而，点文件被火花忽略了。我怎样才能阅读它们？

# This is not loading hidden files into data-frame
dir="/abc/xyz"
df=spark.read.text(dir)

# This is not loading hidden files into data-frame
dir="/abc/xyz/*"
df=spark.read.text(dir)

# This is not loading hidden files into data-frame
dir="/abc/xyz/.*"
df=spark.read.text(dir)

任何建议，将不胜感激。

标签： apache-sparkpysparkhdfs

Answer 1

尝试改变你的路径。

  # This is not loading hidden files into data-frame
    # dir="/abc/xyz/.*"
    dir = "hdfs://yourhost:yourport/abc/xyz/"
    df=spark.read.text(dir)