How to compare the schema of two datasets with Spark?

Asked Jun 18 '19 at 14:18

Active Jun 18 '19 at 14:18

Viewed 195 times

I have two Datasets (dsFile and dsTable) with the following schema :

dsFile :

dsFile
  |-- A: string (nullable = true)
  |-- B: string (nullable = true)
  |-- C: string (nullable = true)

dsTable :

dsTable
  |-- A: string (nullable = true)
  |-- B: string (nullable = true)
  |-- C: string (nullable = true)
  |-- D: string (nullable = true)
  |-- E: string (nullable = true)

I'm wondering if there is a way to find the column differences between this two datasets ? And build a new dsFile dataset with the differences :

dsFileNew :

dsFileNew
  |-- A: string (nullable = true)
  |-- B: string (nullable = true)
  |-- C: string (nullable = true)
  |-- D: string (nullable = true) // column of dsTable
  |-- E: string (nullable = true) // column of dsTable

asked Jun 18 '19 at 14:18

Mamaf

1

possible duplicate of https://stackoverflow.com/questions/47862974/schema-comparison-of-two-dataframes-in-scala? – Zack Jun 18 '19 at 15:15
How do you want to join `dsFile` and `dsTable`? On which column? – Krzysztof Atłasik Jun 18 '19 at 18:28

How to compare the schema of two datasets with Spark?

0 Answers0