python - 如何在 pyspark SQL 中应用日期格式

Question

我们从 pyspark sql 查询下面得到下面的数据

脚本：：

from pyspark.sql import SparkSession
spark = SparkSession.builder.master('yarn').appName('myAppName').getOrCreate()
df = spark.read.parquet('gs://data/test')
df.createOrReplaceTempView("people")

df2=spark.sql("""select id,concat(year(dates) ,'_',month(dates)) as date,
    count(1) count
    from people
group by id, month(dates),year(dates)

预期输出为

比如 2019_jan, 2019_feb, 2019_oct,.... 2019_Dec 请帮我了解 pyspark sql 中的日期格式语法

标签： pythonapache-sparkpysparkpyspark-sql

Answer 1

您可以尝试如下

spark.sql("select date_format(current_date(),'MMM') as month from data")

您的完整查询应该是这样的

from pyspark.sql.functions import *

df2=spark.sql("""select id,concat(year(dates) ,'_',date_format(dates,'MMM')) as date,
    count(1) count from people group by id, date_format(dates,'MMM'),year(dates)