I have three dataframes with specified schemas. Before joining all columns are nullable. However, after a (left outer) join, even columns which do not have udf applied on them, change their nullability. And I don't see any logic how they change it. Not all columns become not nullable. Did someone encounter this?
Basically, I do something like this:
val joinedDf = broadcast (df1).join(df2, col("idInDf1")===col("idInDf2"), "left_outer").join(df3, col("idInDf1")===col("idInDf3),"left_outer")