-3
| date|
+----------+
|  2/3/1994|
|  3/4/1994|
|  4/5/1994|
|  5/3/1994|
|  6/9/1994|
|  7/8/1994|
|  8/9/1994|
| 9/10/1994|
|10/10/1994|
| 11/4/1994|
| 12/3/1994|
|  2/4/1996|
|  4/9/1996|
|    5/7/96|
|  6/8/1996|
| 7/10/1996|
| 9/11/1996|
| 10/3/1996|
|  6/2/2000|
|  7/2/2000|


from pyspark.sql.functions import to_date
newdate=df6.withColumn(to_date(df6.date, 'yyyy-MM-dd').alias('dt')).show()

TypeError: to_date() takes 1 positional argument but 2 were given
Nikhil Suthar
  • 2,289
  • 1
  • 6
  • 24
NUKULA AVINASH
  • 5
  • 2

3 Answers3

0

withColumn syntax seems to be wrong. Can you try this:

newdate=df6.withColumn("new_date", to_date("date", 'dd/MM/yyyy')).show()
pissall
  • 7,109
  • 2
  • 25
  • 45
  • it will not work, to_date required date format to parse date, input date are not in dd-MM-yyyy format , it is in dd/MM/yyyy or MM/dd/yyyy – Nikhil Suthar Jan 30 '20 at 06:20
0
>>> from pyspark.sql.functions  import *
>>> df.show()
+----------+
|      date|
+----------+
|  2/3/1994|
|  3/4/1994|
|  4/5/1994|
|  5/3/1994|
|  6/9/1994|
|  7/8/1994|
|  8/9/1994|
| 9/10/1994|
|10/10/1994|
| 11/4/1994|
| 12/3/1994|
|  2/4/1996|
|  4/9/1996|
|    5/7/96|
|  6/8/1996|
| 7/10/1996|
| 9/11/1996|
| 10/3/1996|
|  6/2/2000|
|  7/2/2000|
+----------+

>>> df.withColumn("dt", to_date(col("date"), "MM/dd/yyyy")).show()
+----------+----------+
|      date|        dt|
+----------+----------+
|  2/3/1994|1994-02-03|
|  3/4/1994|1994-03-04|
|  4/5/1994|1994-04-05|
|  5/3/1994|1994-05-03|
|  6/9/1994|1994-06-09|
|  7/8/1994|1994-07-08|
|  8/9/1994|1994-08-09|
| 9/10/1994|1994-09-10|
|10/10/1994|1994-10-10|
| 11/4/1994|1994-11-04|
| 12/3/1994|1994-12-03|
|  2/4/1996|1996-02-04|
|  4/9/1996|1996-04-09|
|    5/7/96|0096-05-07|
|  6/8/1996|1996-06-08|
| 7/10/1996|1996-07-10|
| 9/11/1996|1996-09-11|
| 10/3/1996|1996-10-03|
|  6/2/2000|2000-06-02|
|  7/2/2000|2000-07-02|
+----------+----------+
Nikhil Suthar
  • 2,289
  • 1
  • 6
  • 24
0

to_date went through a revamp from Spark 2.2.0 onwards, if you are using Spark <2.2.0 then it can take only one argument.

Refer Spark 2.2.0 pyspark.sql.functions.to_date and Spark 2.1.0 pyspark.sql.functions.to_date

Sunny Shukla
  • 342
  • 2
  • 8