pyspark dataframe : TypeError : to_date() takes exactly 1 argument (2 given)

Question

Can someone help to resolve below issue :

I am trying to convert the string to date format in pyspark

datadf1 = datadf1.withColumn('vehicle_last_service_date_temp', to_date(datadf1.vehicle_last_service_date,"mm/dd/yyyy"))

TypeError: to_date() takes exactly 1 argument (2 given)

Possible duplicate of [Convert pyspark string to date format](https://stackoverflow.com/questions/38080748/convert-pyspark-string-to-date-format) — Suresh, Mar 13 '18 at 07:52
Suresh, the same code is working in pyspark console but it is breaking when i am running at aws glue console for pyspark — Yashwant Chandrakar, Mar 13 '18 at 07:57

score 1 · Answer 1 · edited Mar 15 '18 at 18:57

I am able to solve the above mention problem by using unix_timestamp:

import pyspark.sql.functions as F

datadf1 = datadf1.withColumn("vehicle_last_service_date_temp", 
                             F.date_format(F.to_date(F.unix_timestamp(datadf1.vehicle_last_service_date, "mm/dd/yyyy").cast("timestamp")), 'yyyy-MM-dd'))

As far as AWS glue issue for TypeError : to_date() takes exactly 1 argument (2 given).

There is must pyspark version issue, which glue is calling at the back end

pyspark dataframe : TypeError : to_date() takes exactly 1 argument (2 given)

1 Answers1