How delta table stores data in adls gen 2 , lets say if we have 100 versions and each version has update only on few records does that mean it will save entire snapshot. lets say if current/latest snapshot is 10 mb and 20 records gets updated so it will again store 10 mb or only size of 10 records.
Asked
Active
Viewed 57 times
0
-
what are you aiming for? – DileeprajnarayanThumula Aug 09 '23 at 10:19
-
@DileeprajnarayanThumula My delta table size in adls is showing in 2 tb where as latest snapshot is having only half million records. want to understand how size is growing even though i am running vaccum command. Is is like every version is having same size of data or only delta from previous versions. – Ishan Razdan Aug 10 '23 at 05:47
-
have you also tried using the OPTIMIZE command try to compact the files more (default size is 1 gb) and then run vacuum. – DileeprajnarayanThumula Aug 30 '23 at 07:02