I need to transform and union 20ish parquet files (with different schema) into one big fat dataframe in Azure Synapse notebook. I didn't run into any issues when querying each individual dataframe. However, after they are unioned into one table, i've got Data Skew warning message when querying against it. It's not stopping the operation though, so i can still do display(), count rows or even write to adsls, but am just wondering if I need to be concerned about this, and if there's any way to fix it. Thankyou!
Asked
Active
Viewed 67 times
