arrays - Spark：从 Scala 中的嵌套数组中删除第一个数组

Question

我有一个包含 2 列的 DataFrame。我想删除每条记录中嵌套数组的第一个数组。示例：-我有一个像这样的 DF

+---+-------+--------+-----------+-------------+
|id |arrayField                                |
+---+------------------------------------------+
|1  |[[Akash,Kunal],[Sonu,Monu],[Ravi,Kishan]] |
|2  |[[Kunal, Mrinal],[Priya,Diya]]            |
|3  |[[Adi,Sadi]]                              |
+---+-------+---------+----------+-------------+

我想要这样的输出：-

+---+-------+------+------+-------+
|id |arrayField                   |
+---+-----------------------------+
|1  |[[Sonu,Monu],[Ravi,Kishan]]  |
|2  |[[Priya,Diya]]               |
|3  | null                        |
+---+-------+------+------+-------+

标签： arraysscaladataframeapache-sparknested

Answer 1

从Spark-2.4使用slice函数。

Example:

df.show(10,false)
/*
+------------------------+
|arrayField              |
+------------------------+
|[[A, k], [s, m], [R, k]]|
|[[k, M], [c, z]]        |
|[[A, b]]                |
+------------------------+
*/

import org.apache.spark.sql.functions._

df.withColumn("sliced",expr("slice(arrayField,2,size(arrayField))")).
withColumn("arrayField",when(size(col("sliced"))==0,lit(null)).otherwise(col("sliced"))).
drop("sliced").
show()
/*
+----------------+
|      arrayField|
+----------------+
|[[s, m], [R, k]]|
|        [[c, z]]|
|            null|
+----------------+
*/