0

I am new to Spark and have one requirement to remove the key from nested map.

 root
 |-- property: map (nullable = true)
 |    |-- name: string
 |    |-- address: map (valueContainsNull = true)
 |    |    |-- State: string
 |    |    |-- pincode: string (valueContainsNull = true)

Sample code:

val unwantedKey : String = "State"

def updateMap(unwantedKey : String) = udf((colMapName:Map[String,Map[String, String]]) => {colMapName.-(unwantedKey)})

val FinalDF = df2.withColumn("address",updateMap(unwantedKey)(col("address")))

Ankush Singh
  • 1
  • 1
  • Hi! Can you add an example showcasing the sample input and the output you're expecting? Always remember to produce a [mcve] which will help you get better and faster responses. – philantrovert Nov 22 '22 at 09:37

0 Answers0