json - Spark 在列中拆分和解析 json

Question

我有一个 PySpark 数据框：

catalogid   | 1123798                                                                                                                                                                                                                                                                  
catalogpath | [{"1123798":"Other, poets"},{"1112194":"   Poetry for kids"}

使用架构：

StructType(List(StructField(catalogid,StringType,true),StructField(catalogpath,StringType,true)))

而且我只需要从目录路径列中获取文本（值） - 如下所示：

catalogid   | 1123798                                                                                                                                                                                                                                                                  
catalog_desc| "Other, poets"; "Poetry for kids"

标签： jsonapache-sparksplitpyspark

Answer 1

您可以使用 JSON 解析器：

import json
from itertools import chain
from pyspark.sql.functions import udf, concat_ws


@udf("array<string>")
def parse(s):
    try:
        return list(chain.from_iterable(x.values() for x in json.loads(s)))
    except:
        pass

df = spark.createDataFrame(
    [(1123798, """[{"1123798":"Other, poets"},{"1112194":"   Poetry for kids"}]""")],
    ("catalogid", "catalogpath")
)

result = df.select("catalogid", parse("catalogpath").alias("catalog_desc"))

result.show(truncate=False)
# +---------+----------------------------------+
# |catalogid|catalog_desc                      |
# +---------+----------------------------------+
# |1123798  |[Other, poets,    Poetry for kids]|
# +---------+----------------------------------+

如果你想要一个字符串，你可以申请concat_ws：

result.withColumn("catalog_desc", concat_ws(";", "catalog_desc")).show(truncate=False)
# +---------+-------------------------------+
# |catalogid|catalog_desc                   |
# +---------+-------------------------------+
# |1123798  |Other, poets;   Poetry for kids|
# +---------+-------------------------------+