Spark changes column nullability after join

Asked Jun 12 '18 at 15:15

Active Jun 13 '18 at 04:34

Viewed 65 times

I have three dataframes with specified schemas. Before joining all columns are nullable. However, after a (left outer) join, even columns which do not have udf applied on them, change their nullability. And I don't see any logic how they change it. Not all columns become not nullable. Did someone encounter this?

Basically, I do something like this:

val joinedDf = broadcast (df1).join(df2, col("idInDf1")===col("idInDf2"), "left_outer").join(df3, col("idInDf1")===col("idInDf3),"left_outer")

edited Jun 13 '18 at 04:34

asked Jun 12 '18 at 15:15

gorros

1,411
1
18
29

can you share your code snippet? – Kaushal Jun 12 '18 at 21:10
I have added example. – gorros Jun 13 '18 at 04:35
I don't think it is duplicate. I initially have dataframes, and join also as dataframes. I would suspect that `broadcast` or other function affect,but there are columns which are not affected by any. And still become not nullable. – gorros Jun 13 '18 at 04:38
Can you add some input/output example as well as the schema before and after? – Shaido Jun 13 '18 at 04:56

Spark changes column nullability after join

0 Answers0