I had a doubt regarding the position of update_preimage and update_postimage in the table which contains the row_level changes in CDF . Does every update_preimage has its update_postimage right below it i.e. are they in two adjacent rows?
The reason I am asking this is because I wanted to update the sink table with the updates in the source table so I was thinking of bringing both the preimage and the postimage in the same row (maybe by using some window function) and writing a MERGE statement with preimage as the matching condition .
I tried doing the above by using a 'correaled scalar subquery' in the MERGE statement but Spark SQL has some restrictions on it.
