时间戳

首页 > 解决方案 > Convert RDD[(K,V) to Map[K,List[V]]

scala - Convert RDD[(K,V) to Map[K,List[V]]

问题描述

How can i convert a RDD of tuple2 (Key,Value) with duplicate Keys into a Map[K,List[V]] ?

Input example:

val list = List((1,a),(1,b),(2,c),(2,d))
val rdd = sparkContext.parallelize(list)

Output expected:

Map((1,List(a,b)),(2,List(c,d)))

标签： scalardd

解决方案

Just use groupByKey, then collectAsMap:

val rdd = sc.parallelize(List((1,"a"),(1,"b"),(2,"c"),(2,"d")))

rdd.groupByKey.collectAsMap
// res1: scala.collection.Map[Int,Iterable[String]] =
//   Map(2 -> CompactBuffer(c, d), 1 -> CompactBuffer(a, b))

Alternatively, use map/reduceByKey then collectAsMap:

rdd.map{ case (k, v) => (k, Seq(v)) }.reduceByKey(_ ++ _).
  collectAsMap
// res2: scala.collection.Map[Int,Seq[String]] =
//   Map(2 -> List(c, d), 1 -> List(a, b))

推荐阅读