scala - 使用 Spark 中的地图将日期转换为字符串不起作用？

Question

我正在使用 Spark 2.4.5。

我写了以下代码：

val df_join: DataFrame = orders
      .join(order_items, orders.col(("order_id")) === order_items.col("order_item_order_id"), "inner")
      .withColumn("order_date", to_date(from_unixtime(col("order_date")/1000), "yyyy-MM-dd"))

df.show(false)

+-----------------+----------+--------+---------------+-------------+-------------------+---------------------+------------------------+-------------------+-------------------+
|order_customer_id|order_date|order_id|order_status   |order_item_id|order_item_order_id|order_item_product_id|order_item_product_price|order_item_quantity|order_item_subtotal|
+-----------------+----------+--------+---------------+-------------+-------------------+---------------------+------------------------+-------------------+-------------------+
|12143                     |2013-11-09|17210   |PENDING_PAYMENT|43051        |17210              |627                  |39.99                   |1                  |39.99              |
|1595                       |2013-11-09|17211   |COMPLETE       |43052        |17211              |1014                 |49.98                   |3                  |149.94             |
|1595                       |2013-11-09|17211   |COMPLETE       |43053        |17211              |191                  |99.99                   |5   


df_join.printSchema

root
 |-- order_customer_id: integer (nullable = true)
 |-- order_date: date (nullable = true)
 |-- order_id: integer (nullable = true)
 |-- order_status: string (nullable = true)
 |-- order_item_id: integer (nullable = true)
 |-- order_item_order_id: integer (nullable = true)
 |-- order_item_product_id: integer (nullable = true)
 |-- order_item_product_price: float (nullable = true)
 |-- order_item_quantity: integer (nullable = true)
 |-- order_item_subtotal: float (nullable = true)

I tried this :  

var comByKeyResult: Dataset[((String, String), (Double, Int))] = df_join
      .map(x => ((x(1).toString, x(3).toString), (x(9).toString.toDouble, x(1).toString.toInt)))

当我尝试这样做时：comByKeyResult.take(5).foreach(println)

我有以下错误，我不明白原因：

java.lang.NumberFormatException: For input string: "2013-11-09"

为什么.map(x => ((x(1).toString不起作用order_date: date (nullable = true)？

标签： scalaapache-spark