4

We are comparing Change Data Capture (CDC) capabilities for AWS Glue to SnapLogic and Informatica. AWS Glue has the ability to detect changes in the data structure.

I am looking for specific examples of how to detect changes in data (i.e. modified data or new data). Has someone used AWS Glue to pull in only new/modified records? If so, how?

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
Miles Baker
  • 53
  • 1
  • 3

1 Answers1

0

The two option to achieve CDC in glue is by 1. using audit column in the source database and passing it in the sql to extract data 2. If the data is no more than few hundred thousand records then extract the full data and compare using spark sql.

Abraham
  • 423
  • 3
  • 9
  • Could you elaborate on the first method. How does one go about achieving it. Is there any online tutorial available? Thanks – axiom Feb 03 '20 at 08:29