Spark wrongly casting integers as `struct`

Question

In a spark job, I am using

.withColumn("year", year(to_timestamp(lit(col("timestamp")))))

This code used to work. But now I get the error :

"cannot resolve 'CAST(`timestamp` AS TIMESTAMP)' due to data type mismatch: cannot cast struct<int:int,long:bigint> to timestamp;"

I looks like spark is reading my timestamp column as a struct<int:int,long:bigint> instead of a int

How can I prevent that ?

Context the initial data is in jsonline. I read it using AWS GLUE glueContext.create_dynamic_frame.from_catalog. In the GLUE catalog the timestamp column is typed int.

I found a way to force the type of timestamp before parsing it — Hugo, May 19 '21 at 20:53

score 2 · Answer 1 · answered May 19 '21 at 20:49

2

Finally I solved it this way :

GF_resolved = ResolveChoice.apply(
    frame=GF_raw,
    specs=[("timestamp", "cast:int")],
    transformation_ctx="resolve timestamp type",
)

ResolveChoice is method avaible on AWS Glue DynamicFrame

answered May 19 '21 at 20:49

Hugo

1,195
2
12
36

jpizzo · Answer 2 · 2022-07-14T16:42:01.357

0

The short answer is that you cannot prevent it if creating a dynamic frame from catalog because, as the name suggests, the schema is dynamic. See this SO for more information.

Alternative approach that is a little more compact is...

gf_resolved = gf_raw.resolveChoice(specs = [('timestamp','cast:int')])

Official documentation for the resolve choice class can be found here. AWS Resolve Choice

edited Jul 14 '22 at 16:42

answered Jul 14 '22 at 16:35

jpizzo

143
2
7

Spark wrongly casting integers as `struct`

2 Answers2