I have a database implemented with Databricks Delta Lake and Azure Storage. I need to create a table that would have a new data ingested on daily basis. The data would come from diffrent sources. Each day the volume can differ (it can be minor or vast data ingestion). My question is: how should I partition this table?
In the past I had my tables partitioned by ingestion date.
