pyspark - 在 Pyspark [非熊猫 df] 中将许多数据帧合并为一个
问题描述
我将通过一个过程一一生成数据帧。我必须将它们合并为一个。
+--------+----------+
| Name|Age |
+--------+----------+
|Alex | 30|
+--------+----------+
+--------+----------+
| Name|Age |
+--------+----------+
|Earl | 32|
+--------+----------+
+--------+----------+
| Name|Age |
+--------+----------+
|Jane | 15|
+--------+----------+
最后:
+--------+----------+
| Name|Age |
+--------+----------+
|Alex | 30|
+--------+----------+
|Earl | 32|
+--------+----------+
|Jane | 15|
+--------+----------+
尝试了许多选项,例如 concat、merge、append,但我猜都是 pandas 库。我没有使用熊猫。使用版本 python 2.7 和 Spark 2.2
编辑以使用 foreachpartition 覆盖最终场景:
l = [('Alex', 30)]
k = [('Earl', 32)]
ldf = spark.createDataFrame(l, ('Name', 'Age'))
ldf = spark.createDataFrame(k, ('Name', 'Age'))
# option 1:
union_df(ldf).show()
#option 2:
uxdf = union_df(ldf)
uxdf.show()
两种情况下的输出:
+-------+---+
| Name|Age|
+-------+---+
|Earl | 32|
+-------+---+
解决方案
您可以unionAll()
用于数据框:
from functools import reduce # For Python 3.x
from pyspark.sql import DataFrame
def unionAll(*dfs):
return reduce(DataFrame.union, dfs)
df1 = sqlContext.createDataFrame([(1, "foo1"), (2, "bar1")], ("k", "v"))
df2 = sqlContext.createDataFrame([(3, "foo2"), (4, "bar2")], ("k", "v"))
df3 = sqlContext.createDataFrame([(5, "foo3"), (6, "bar3")], ("k", "v"))
unionAll(df1, df2, df3).show()
## +---+----+
## | k| v|
## +---+----+
## | 1|foo1|
## | 2|bar1|
## | 3|foo2|
## | 4|bar2|
## | 5|foo3|
## | 6|bar3|
## +---+----+
编辑:
您可以创建一个空数据框,并继续对其进行联合:
# Create first dataframe
ldf = spark.createDataFrame(l, ["Name", "Age"])
ldf.show()
# Save it's schema
schema = ldf.schema
# Create an empty DF with the same schema, (you need to provide schema to create empty dataframe)
empty_df = spark.createDataFrame(spark.sparkContext.emptyRDD(), schema)
empty_df.show()
# Union the first DF with the empty df
empty_df = empty_df.union(ldf)
empty_df.show()
# New dataframe after some operations
ldf = spark.createDataFrame(k, schema)
# Union with the empty_df again
empty_df = empty_df.union(ldf)
empty_df.show()
# First DF ldf
+----+---+
|Name|Age|
+----+---+
|Alex| 30|
+----+---+
# Empty dataframe empty_df
+----+---+
|Name|Age|
+----+---+
+----+---+
# After first union empty_df.union(ldf)
+----+---+
|Name|Age|
+----+---+
|Alex| 30|
+----+---+
# After second union with new ldf
+----+---+
|Name|Age|
+----+---+
|Alex| 30|
|Earl| 32|
+----+---+
推荐阅读
- php - PHP - 例如,“:bool”在PHP函数之后究竟是什么意思?
- javascript - 单击标签时在 div 中加载 PHP 文件
- range - 如何使用偏移变量选择多个单元格但保持前 2 行中的内容被选中
- ios - 推送通知到达时获取数据 Swift 4.2
- python - 在 python 中使用 meshlab 函数
- javascript - 带有模块和主要转译的 Webpack
- java - 尝试使用 fetch 类型的惰性序列化实体
- java - 如何在 Vertx 页面上从 MariaDB 获取和显示数据?
- r - 根据 hcharter 中的给定序列设置热图颜色范围
- python - 管道几个熊猫数据框