0

I have a use case where the table columns will be changing [addition/deletion] at each refresh [currently its weekly refresh]. It is stored as delta format. is there any way that can we trach the version of these column addition/deletion like a kind of meta store.

Is there any where I can find such information in delta table or delta file format ?

Alex Ott
  • 80,552
  • 8
  • 87
  • 132
MJ029
  • 151
  • 1
  • 12
  • how do you write the data? using the `.mode("overwrite")` ? – Alex Ott Feb 17 '22 at 13:25
  • Hi @AlexOtt, Actually I'm using .mode("append"). it is transactional data. – MJ029 Feb 17 '22 at 13:27
  • you mean that you're changing the data? I don't understand the question completely – Alex Ott Feb 17 '22 at 13:36
  • Hi @AlexOtt, Im basically using databricks delta format for one of my table where it is transactional data, so i need to append on weekly basis but one cache here is that the number of columns that we receive for every week is purely dynamic [20% of columns]. If i could able to get the versions history from delta table where I can find since when the column is ingested and available in delta table, that would be more helpful for me to write dynamic query based on the version – MJ029 Feb 17 '22 at 13:38

0 Answers0