apache-spark - 如何在 Spark2 中将 Struct 类型的 Map 转换为 Json

Question

我在数据集中有一个地图字段，其架构如下

     |-- party: map (nullable = true)
     |    |-- key: string
     |    |-- value: struct (valueContainsNull = true)
     |    |    |-- partyName: string (nullable = true)
     |    |    |-- cdrId: string (nullable = true)
     |    |    |-- legalEntityId: string (nullable = true)
     |    |    |-- customPartyId: string (nullable = true)
     |    |    |-- partyIdScheme: string (nullable = true)
     |    |    |-- customPartyIdScheme: string (nullable = true)
     |    |    |-- bdrId: string (nullable = true)

需要将其转换为 JSON 类型。请建议如何做。提前致谢

标签： apache-sparkapache-spark-sqlapache-spark-2.2

Answer 1

Spark 提供了可用于 DataFrame 操作的to_json函数：

import org.apache.spark.sql.functions._
import spark.implicits._
val df =
  List(
    ("key1", "party01", "cdrId01"),
    ("key2", "party02", "cdrId02"),
  )
    .toDF("key", "partyName", "cdrId")
    .select(struct($"key", struct($"partyName", $"cdrId")).as("col1"))
    .agg(map_from_entries(collect_set($"col1")).as("map_col"))
    .select($"map_col", to_json($"map_col").as("json_col"))