scala - 如何在 Spark Scala 的 For 循环中动态创建多个数据帧

Question

我有一个 Array[string] ，其值为 ["DF1" , "DF2", "DF3"]

我的配置文件中有如下值。

DF1=select * from TB1
DF2=select * from TB2
DF3=select * from TB3

我需要通过卸载相应的表在 For 循环中动态创建 3 个数据框。

假设我的数组名称是 ARRAY1。

我的代码是

for (for1 <- ARRAY1){val $for1+_DF = spark.sql(for1)}

以上只是一种伪代码。

请通过提供正确的代码语法来帮助我。

谢谢，纳文

标签： scalaapache-spark

Answer 1

检查下面的代码。

scala> val queries = Map(
                          "df1" -> "select * from tb1",
                          "df2" -> "select * from TB2",
                          "df3" -> "select * from TB3"
                      ) // After reading from config file.

scala> 

queries
.values
.par
.map(spark.sql)
.foreach(_.show(false)) 

// Used .par for parallel loading & all three DataFrame object will be in list & you can do any operation on that, here i am using show function to display values

+---+
|id |
+---+
|1  |
|2  |
|3  |
|4  |
|5  |
|6  |
|7  |
|8  |
|9  |
+---+

+---+
|id |
+---+
|1  |
|2  |
|3  |
|4  |
|5  |
|6  |
|7  |
|8  |
|9  |
+---+

+---+
|id |
+---+
|1  |
|2  |
|3  |
|4  |
|5  |
|6  |
|7  |
|8  |
|9  |
+---+