0

In my old Spark2.X code, I had a following line

pageviewsDF.groupBy( date_format(col("capturedAt"), "u-E").alias("Day Of Week") ).sum('req')

That will give Day of Week as 1-Mon, 2-Tue etc.

But now in Spark3 I get an error that u-E not recognised and I can use legacy setting as below

spark.conf.set("spark.sql.legacy.timeParserPolicy","LEGACY")

But is there a way to get same output without using legacy setting?

chintan s
  • 6,170
  • 16
  • 53
  • 86

1 Answers1

0

You can use the expr expression to generate the desired format.

import pyspark.sql.functions as F

...
pageviewsDF.groupBy(F.expr('concat(dayofweek(capturedAt), "-", date_format(capturedAt, "E"))').alias("Day Of Week")).sum('req')
过过招
  • 3,722
  • 2
  • 4
  • 11