I have parquet files stored in ADLS gen2 with such structure:
year/month/day
part_*.snappy.parquet
The files in the folders represent the same dataset, but which schema is changing ("evolving") over time. So, for 2023-01-15 schema of the parquet file could be different compared to the schema from the file from the 2023-05-01 folder.
I would like to convert this parquet files to Delta format, so I'm trying this:
convert to delta parquet.`abfss://container@storageAccount.dfs.core.windows.net/parquet-data`;
But I'm getting an error "Failed to merge schema of file".
What are possible options to converting such data to Delta?
