0

I have three dataframes with specified schemas. Before joining all columns are nullable. However, after a (left outer) join, even columns which do not have udf applied on them, change their nullability. And I don't see any logic how they change it. Not all columns become not nullable. Did someone encounter this?

Basically, I do something like this:

val joinedDf = broadcast (df1).join(df2, col("idInDf1")===col("idInDf2"), "left_outer").join(df3, col("idInDf1")===col("idInDf3),"left_outer")
gorros
  • 1,411
  • 1
  • 18
  • 29

0 Answers0