0

I have an SQL which I am trying to code in spark scala using dataframes

SELECT country, 
       Substr(substring_col, 1, 3) AS Code, 
       CASE 
         WHEN Substr(substring_col, 1, 9) = '238208700' THEN 
         'columnName1' 
         WHEN Substr(substring_col, 1, 9) = '240018000' THEN 'columnName2' 
         WHEN Substr(substring_col, 1, 9) = '240017531' THEN 'columnName3' 
         WHEN Substr(substring_col, 1, 9) = '240017001' 
               OR Substr(substring_col, 1, 9) = '240017301' 
               OR Substr(substring_col, 1, 9) = '240017302' THEN 'columnName4' 
         WHEN Substr(substring_col, 1, 9) = '240017211' THEN 'columnName5' 
         WHEN Substr(substring_col, 1, 9) = '248010160' 
               OR Substr(substring_col, 1, 9) = '248010241' 
               OR Substr(substring_col, 1, 9) = '248010420' THEN 'columnName6' 
         ELSE 'not_defined' 
       END                AS custom_column_name 
FROM   t_filtered 

Below is code which I am trying to make it look pretty but unable to achieve the desired result

val substring_col = substring(col("col_name"),1,9)
val mapper = Map("columnName1"  -> List("238208700"),
  "columnName2" -> List("240017301", "240017001")
)

myDf.withColumn("custom_column_name" ,mapper.foldLeft(lit(""))((accu, mapperMap) => {
  when(substring_col isin mapperMap._2, mapperMap._1)
}))

I get the below error

Unsupported literal type class scala.collection.immutable.$colon$colon List(238208700)
java.lang.RuntimeException: Unsupported literal type class scala.collection.immutable.$colon$colon List(238208700)
Gladiator
  • 354
  • 3
  • 19

0 Answers0