0

I have a configuration in a form of a map:

val config =  Map[String, Array[String]] = Map("column1" -> Array("field1"), "column2" -> Array("field1","field2"))

I want to use foldLeft to apply this to a dataframe and dynamically filter nested fields using dropFields functions.

I came out with:

val result = config.foldLeft(""){(k, v) =>
  df.withColumn( v._1, col(k + v._1).dropFields(v._2: _*))
}

but struggle to make foldleft work, any help would be appreciated.

Kylo
  • 109
  • 8
  • What do you think `k` is supposed to be? – Dima May 04 '22 at 12:25
  • Its just stuck me, it should be DataFrame, which I should pass in foldLeft – Kylo May 04 '22 at 12:28
  • Looks like this is working : `val result = excludedNestedColumns.foldLeft(sourceDF){(df, v) => df.withColumn(v._1, col(v._1).dropFields(v._2: _*)) }` – Kylo May 04 '22 at 12:41

0 Answers0