am tring to get weekdays from given timestamp rquired to use to_date but returns null

Question

Hi i'm trying to get weekday from timestamp with am pm. so I have a dataframe

dataframe = spark.createDataFrame(

        data = [ ("1","9/8/2019 10:01:28 PM")],

        schema=["id","input_timestamp"])
dataframe.show()
dataframe.printSchema()

And if I do this its giving null as output. I have to do it using to_date

dataframe.withColumn("timestamp",to_date("input_timestamp", "MM/dd/yyyy HH:mm:ss am")) \
.show(truncate=False)

+---+--------------------+---------+
|id |input_timestamp     |timestamp|
+---+--------------------+---------+
|1  |9/8/2019 10:01:28 PM|null     |
+---+--------------------+---------+

Answer 1

Use M/d/yyyy hh:mm:ss a as format.

spark.conf.set('spark.sql.legacy.timeParserPolicy', 'LEGACY')

spark.sparkContext.parallelize([("9/8/2019 10:01:28 PM",)]).toDF(['ts_str']). \
    withColumn('ts', func.to_timestamp('ts_str', 'M/d/yyyy hh:mm:ss a')). \
    show(truncate=False)

# +--------------------+-------------------+
# |ts_str              |ts                 |
# +--------------------+-------------------+
# |9/8/2019 10:01:28 PM|2019-09-08 22:01:28|
# +--------------------+-------------------+

# root
#  |-- ts_str: string (nullable = true)
#  |-- ts: timestamp (nullable = true)