1

I have a column in pyspark dataframe which is in the format 2021-10-28T22:19:03.0030059Z (string datatype). How to convert this into a timestamp datatype in pyspark?

I'm using the code snippet below but this returns nulls, as it's unable to convert it. Can someone please recommend on how to convert this?

df3.select(to_timestamp(df.DateTime, 'yyyy-MM-ddHH:mm:ss:SSS').alias('dt'),col('DateTime')).show()
Adrian Mole
  • 49,934
  • 160
  • 51
  • 83
thedataguy
  • 11
  • 4
  • to_timestamp can parse only till Seconds and not milli seconds. Below post has different ways of handling these millisecs but the easist way what worked for me to use the below code. https://stackoverflow.com/questions/54044012/pyspark-to-timestamp-does-not-include-milliseconds/54192293. df = df.withColumn('time', df.time.cast("timestamp")) – thedataguy Nov 07 '21 at 22:22

0 Answers0