scala - 如何在 spark scala 中获得加入 RDD 中的特定字段 val？

Question

人口普查（Id：String，Emptype：String，Salary：Int） 所得税（EMptype：String，Slab：Int）

加入这两个 RDD 后。如何过滤工资大于 50,000 的值以及连接表中的其他字段？（在火花斯卡拉）

标签： scalaapache-spark

Answer 1

使用filter或where过滤来自 DataFrame 的数据。检查下面的代码。

scala> case class Census(id:String, emptype:String, salary:Int)
defined class Census

scala> case class Incometax(emptype:String,slab:Int)
defined class Incometax

scala> val censusDF = Seq(Census("1","a",10000),Census("2","b",20000),Census("3","c",60000)).toDF
censusDF: org.apache.spark.sql.DataFrame = [id: string, emptype: string ... 1 more field]

scala> val incometaxDF = Seq(Incometax("a",10),Incometax("b",15),Incometax("c",20)).toDF
incometaxDF: org.apache.spark.sql.DataFrame = [emptype: string, slab: int]

scala> censusDF.join(incometaxDF,Seq("emptype"),"left").filter(censusDF("salary") > 50000).show(false)
+-------+---+------+----+
|emptype|id |salary|slab|
+-------+---+------+----+
|c      |3  |60000 |20  |
+-------+---+------+----+

scala> censusDF.join(incometaxDF,Seq("emptype"),"left").where(censusDF("salary") > 50000).show(false)
+-------+---+------+----+
|emptype|id |salary|slab|
+-------+---+------+----+
|c      |3  |60000 |20  |
+-------+---+------+----+