apache-spark - 如何将参数传递给 spark.sql(""" """)？

Question

我想将一个字符串传递给 spark.sql

这是我的查询

mydf = spark.sql("SELECT * FROM MYTABLE WHERE TIMESTAMP BETWEEN '2020-04-01' AND '2020-04-08') 

我想为日期传递一个字符串。

我试过这段代码

val = '2020-04-08'

s"spark.sql("SELECT * FROM MYTABLE WHERE TIMESTAMP  BETWEEN $val  AND '2020-04-08'

标签： apache-sparkpysparkapache-spark-sqlapache-zeppelin

Answer 1

尝试使用 Python 字符串格式，{}并在 scala 中使用。.format(val)$val

val = '2020-04-08'

spark.sql("SELECT * FROM MYTABLE WHERE TIMESTAMP  BETWEEN {}  AND '2020-04-08'".format(val)).show()

---

Example:

In Pyspark:

spark.sql("select * from tmp").show()
#+----+---+
#|name| id|
#+----+---+
#|   a|  1|
#|   b|  2|
#+----+---+

id='1'

spark.sql("select * from tmp where id={}".format(id)).show()
#+----+---+
#|name| id|
#+----+---+
#|   a|  1|
#+----+---+

---

In Scala:

用于string interpolation替换变量的值

val id=1
spark.sql(s"select * from tmp where id=$id").show()
//+----+---+
//|name| id|
//+----+---+
//|   a|  1|
//+----+---+