I am trying to translate a pyspark job, which is dynamically coalescing the columns from two datasets with additional filters/condition.
conditions_ = [when(df1[c]!=df2[c], lit(c)).otherwise("") for c in df1.columns if c not in ['firstname','middlename','lastname']]
can I do this in scala?
What I have tried so far is:
df1.join(df2, Seq("col1"), "outer").select(col("col1"), coalesce(df1.col("col2"), df2.col("col2")).as(col("col2"), coalesce(df1.col("col3")..........as(col("col30"))
is there a better way to add them with a loop instead of expanding this?