Spark parse string to timestamp with timezone

Question

I have a string like:

2018-03-21T08:15:00+01:00

and wonder how to preserve the time zone / shift from UTC when parsing it in Spark.

Seq("2018-03-21T08:15:00+01:00").toDF.select('value, to_timestamp('value, "yyy-MM-ddTHH:mm:ss")).show(false)

unfortunately only yields null. Even my format string which is omitting the shift only returns null.

score 8 · Accepted Answer · answered Apr 01 '18 at 14:19

8

T is not a format specifier so it should be escaped:

"yyyy-MM-dd'T'HH:mm:ss"

and timezone is denoted by X

"yyy-MM-dd'T'HH:mm:ssXXX"

answered Apr 01 '18 at 14:19

Alper t. Turker

1 Answers1