Error while exploding a struct column in Spark

Question

I have a dataframe whose schema looks like this:

event: struct (nullable = true)
|    | event_category: string (nullable = true)
|    | event_name: string (nullable = true)
|    | properties: struct (nullable = true)
|    |    | ErrorCode: string (nullable = true)
|    |    | ErrorDescription: string (nullable = true)

I am trying to explode the struct column properties using the following code:

df_json.withColumn("event_properties", explode($"event.properties"))

But it is throwing the following exception:

cannot resolve 'explode(`event`.`properties`)' due to data type mismatch: 
input to function explode should be array or map type, 
not StructType(StructField(IDFA,StringType,true),

How to explode the column properties?

@user8371915 As that question has been marked as a duplicate of this question your close vote would now cause cyclical duplicate navigation (and isn't valid if tried now) — Nick is tired, Jan 19 '18 at 15:49

score 12 · Accepted Answer · answered Jan 18 '18 at 09:48

12

You can use explode in an array or map columns so you need to convert the properties struct to array and then apply the explode function as below

import org.apache.spark.sql.functions._
df_json.withColumn("event_properties", explode(array($"event.properties.*"))).show(false)

You should have your desired requirement

answered Jan 18 '18 at 09:48

Ramesh Maharjan

41,071
6
69
97

3

how can explode as Map. I need key name too – shiva.n404 Jan 18 '18 at 18:08
what is the array function you use? Is this the pyspark ArrayType data type function? or is this a sql func usually imported as F.? – Artem Yevtushenko Nov 15 '22 at 15:43
array is the spark sql functions that I imported – Ramesh Maharjan Dec 30 '22 at 09:06

score 8 · Answer 2 · answered Jan 18 '18 at 08:12

as the error message says, you can only explode array or map types, not struct type columns.

You can just do

df_json.withColumn("event_properties", $"event.properties")

This will generate a new column event_properties, which is also of struct-type

If you want to convert every element of the struct to a new column, then you cannot use withColumn, you need to do a select with a wildcard *:

df_json.select($"event.properties.*")

score 0 · Answer 3 · answered Jan 18 '18 at 08:16

You may use following to flatten the struct. Explode does not work for struct as error message states.

val explodeDF = parquetDF.explode($"event") { 
case Row(properties: Seq[Row]) => properties.map{ property =>
  val errorCode = property(0).asInstanceOf[String]
  val errorDescription = property(1).asInstanceOf[String]
  Event(errorCode, errorDescription, email, salary)
 }
}.cache()
display(explodeDF)

Error while exploding a struct column in Spark

3 Answers3

Linked

Related