python - 如何将火花流保存到本地 PC 和 hdfs?
问题描述
尝试此数据正在流式传输,并且无法以元组的形式将该数据保存在本地磁盘或 hdfs 中。从 pyspark 导入 SparkConf,SparkContext
from operator import add
import sys
from pyspark.streaming import StreamingContext
from pyspark.streaming.kafka import KafkaUtils
## Constants
APP_NAME = "PythonStreamingDirectKafkaWordCount"
##OTHER FUNCTIONS/CLASSES
def main():
sc = SparkContext(appName="PythonStreamingDirectKafkaWordCount")
ssc = StreamingContext(sc, 2)
brokers, topic = sys.argv[1:]
kvs = KafkaUtils.createDirectStream(ssc, [topic], {"metadata.broker.list": brokers})
lines = kvs.map(lambda x: x[1])
counts = lines.flatMap(lambda line: line.split(" ")) \
.map(lambda word: (word, 1)) \
.reduceByKey(lambda a, b: a+b)
def process(RDD):
#RDD.pprint()
kvs2=RDD.map()
kvs2.saveAsTextFiles('path')
#kvs.foreachRDD(lambda x: process(x))
#kvs1=kvs.map(lambda x: x)
kvs.pprint()
kvs.saveAsTextFiles('path','txt')
ssc.start()
ssc.awaitTermination()
if __name__ == "__main__":
main()
解决方案
在这一行:
kvs.saveAsTextFiles('path','txt')
您正在存储原始流,而不是带有元组的流。而是从计数中存储:
counts.saveAsTextFiles('path','txt')
好奇保存在“路径”中提供的目录下的工作节点上的文件。
对于最新版本,pySpark API 不支持保存到 HDFS,其他语言确实有saveAsHadoopFiles。链接到文档。
推荐阅读
- python - Applying map function to OO method call
- windows - find the status of the hotfixes installed in the system using WMI/ Win32
- python - 如何通过某个步骤生成随机数?
- vue.js - Swiper doesn't work correctly when looping (last and first slides are empty)
- javascript - localhost 没有发送任何数据 ERR_EMPTY_RESPONSE
- powershell - 用于远程计算机的 Powershell Get-Volumn
- r - Matching a dataset with the closest neighbour in another dataset
- javascript - 让子函数递归
- object - Delphi编译器错误?使用“对象”,但使用“记录”编译
- r - R: data.table inside GA optimization throws error