Pyspark : How to concat two dataframes in Pyspark

Question

df_1 :

NBB1
776

And df_2

NBB2
4867

I will to obtain this dataframe in Pyspark df :

NBB1	NBB2
776	4867

Does this answer your question? [How to concatenate/append multiple Spark dataframes column wise in Pyspark?](https://stackoverflow.com/questions/44320699/how-to-concatenate-append-multiple-spark-dataframes-column-wise-in-pyspark) — blackbishop, Apr 13 '22 at 10:29

score 3 · Accepted Answer · answered Apr 13 '22 at 11:39

You need to perform a crossJoin between the two dataframes. See below for details -

from pyspark.sql import Row

df1 = spark.createDataFrame([Row(NBB1 = 776)])
df1.show()
#Output
+----+
|NBB1|
+----+
| 776|
+----+

df2 = spark.createDataFrame([Row(NBB2 = 4867)])
df2.show()
#Output
+----+
|NBB2|
+----+
|4867|
+----+


df1.crossJoin(df2).show()
#Output
+----+----+
|NBB1|NBB2|
+----+----+
| 776|4867|
+----+----+

Pyspark : How to concat two dataframes in Pyspark

1 Answers1