0

I have a date column which is string in dataframe in the 2017-01-01 12:15:43 timestamp format.

Now I want to get weekday number(1 to 7) from that column using dataframe and not spark sql.

Like below

df.select(weekday(col("colname")))

I found one in python and sql but not in scala. can any body help me on this

in sqlcontext

sqlContext.sql("select date_format(to_date('2017-01-01'),'W') as week")
vsminkov
  • 10,912
  • 2
  • 38
  • 50
Babu
  • 861
  • 3
  • 13
  • 36
  • The Scala and Python code will be essentially identical. You could literally copy and paste from one to another (excluding assignments). – Alper t. Turker May 02 '18 at 15:20
  • https://stackoverflow.com/questions/38928919/how-to-get-the-weekday-from-day-of-month-using-pyspark?utm_medium=organic&utm_source=google_rich_qa&utm_campaign=google_rich_qa – Babu May 02 '18 at 15:29
  • I was trying to convert that but not working – Babu May 02 '18 at 15:30
  • These QA's tend to show up in google searches for pyspark solutions. For anyone else who similarly stumbled here. See `pyspark.sql.functions.date_format` and [simple date format](https://docs.oracle.com/javase/8/docs/api/java/text/SimpleDateFormat.html) for format codes. – Wassadamo Sep 27 '21 at 08:35

1 Answers1

5

This works the same way in Scala:

scala> spark.version
res1: String = 2.3.0

scala> spark.sql("select date_format(to_date('2017-01-01'),'W') as week").show
// +----+
// |week|
// +----+
// |   1|
// +----+

or 

scala> import org.apache.spark.sql.functions._
import org.apache.spark.sql.functions._

scala> val df = Seq("2017-01-01").toDF("date")
df: org.apache.spark.sql.DataFrame = [date: string]

scala> df.select(date_format(to_date(col("date")), "W")).show
+-------------------------------+
|date_format(to_date(`date`), W)|
+-------------------------------+
|                              1|
+-------------------------------+
user9731179
  • 66
  • 1