1

We have a DWH that is connected to several sources DB's. We recently faced an issue where one of the sources inserted a new set of records with Timestamp that is in the past (not the actual Timestamp of the insertion to their DB). We use the Timestamp to extract Delta records. So in this case these new set of records are not getting extracted in our delta extraction. I believe using rowversion would be an ideal solution but we do not have control over this source and we can't guarantee this won't happen again. What would be a good solution to handle such cases? We use Datastage.

Thanks!

Meshal
  • 13
  • 4
  • 1
    Ensure that data is never inserted into the databases with timestamps in the past? Seems to be a really bad thing for whoever did it to have done, not least because it breaks the ETL processes! – NickW Sep 14 '21 at 07:05
  • Also ensure that the timestamp you're using is **no business-attribute** at all. It should represent the actual _technical moment_ the data was written or changed the last time. Anything else e.g. like an invoice date is not suitable. _(Although I know that's the most commonly used one. Painful discussions. Always.)_ – Justus Kenklies Nov 14 '21 at 07:00

0 Answers0