I have a table, 665k rows that takes a considerable amount of processing time to generate, with a lot of matching of text fields etc. I have optimized it as far as I can but its still too big to run nightly. Most of this data isn't going to change though and I can pick up the ~600 or so additions overnight, adding the new records and updating existing records. But its the existing records that are the problem.
I need incremental update in the data flow to replace an existing record if the KEY is present and only add the record if the KEY is new. From all I can see, incremental refresh is just going to keep pouring data in. Its frustrating that its not Key agnostic.
Someone else must have come across this problem before?
TIA
Tried playing with incremental refresh on premium service. Didn't work out
