dataframe - Pyspark - 基于列表中的值爆炸数据框

Question

这是我的数据框：

+--------------------+--------------------+
|             core_id|    movie_genres_upd|
+--------------------+--------------------+
|12f99f04-5168-438...|[Comedy, Mockumen...|
|32c7d12f-6bf2-4e5...|[Action, Blockbus...|
|9f067041-3b49-4db...|[Animation, Comed...|
|c6d203cb-afcf-4e8...|[Action, Adventur...|
|b02416f9-5761-48f...|[Adventure, Anima...|

这些是我的数据类型：

[('core_id', 'string'), ('movie_genres_upd', 'array<string>')]

我将提供一个更明显的例子。这是初始数据框：

id  genres
1   ["comedy", "blockbuster"]
2   ["drama", "animation", "comedy"] 

所需的数据框：

id genres
1  "comedy"
1  "blockbuster"
2  "drama"
2  "animation"
2  "comedy"

我是 pyspark 的新手，所以我正在为此苦苦挣扎。任何帮助将非常感激。

标签： dataframepyspark

Answer 1

让我知道这是否有帮助：

>>> from pyspark.sql.functions import explode
>>> from pyspark.sql.types import (
...     StringType,
...     StructField,
...     StructType,
...     ArrayType
... )
>>>
>>> schema = StructType([
...     StructField('core_id', StringType(), True),
...     StructField('movie_genres_upd', ArrayType(StringType()), True)
... ])
>>>
>>> list = [[1, ["comedy", "blockbuster"]], [2, ["drama", "animation", "comedy"]]]
>>> df = spark.createDataFrame(list, schema)
>>> df2 = df.select('core_id', explode("movie_genres_upd").alias('genre'))
>>> df2.show()
+-------+-----------+
|core_id|      genre|
+-------+-----------+
|      1|     comedy|
|      1|blockbuster|
|      2|      drama|
|      2|  animation|
|      2|     comedy|
+-------+-----------+