-1

I am running an incremental process for multiple tables where I am using old watermark, new watermark, copy activity, and a stored procedure to update the watermark value. One of my table succeeds when I run the pipeline, for eg I can see total rows copied as 900 whic matches source SQl but when I check in the Database I can see only 70 rows getting reflected, I am not sure why I am not able to see all the rows at database level. Is this some kind of bug in ADF. Need some advise and suggestions on my situation

Thank You

gmp
  • 1
  • 3
  • Can you check the linked service connections? Whether it is refering to same target database? – Aswin Aug 17 '23 at 09:53
  • Yes it referencing to the same target database – gmp Aug 17 '23 at 09:55
  • What exact test are you using to asceratin that only 70 rows are affected in the target? Are you doing a count on the table? Perhaps it's merging, which means that possibly 830 rows are being updated. This is not a bug in ADF. You'll need to explain in a lot more detail to get to the bottom of it – Nick.Mc Aug 18 '23 at 02:29
  • @Nick.McDermaid, Nick I am almost sure this is because the rows are skipped. Yes its not a bug. – Ziya Mert Karakas Aug 19 '23 at 23:29
  • 1
    I don't think there is any level of sure based on the problem description. – Nick.Mc Aug 20 '23 at 08:00

2 Answers2

0

Its not the first time I am seeing this, its quite common complaint because "rows written" might be confusing to some. Can you implement data consistency verification in copy activity? And what is your fault tolerance?

https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-data-consistency

enter image description here

Some rows may be skipped, in this case for example ADF will still show you like they are "written" because logically they are, but physically not, because of incompatible rows etc. which are skipped. In other words, skipped rows are counted as "written" in ADF terminology. Should check that as well

Ziya Mert Karakas
  • 572
  • 3
  • 15
0

This happens the rows are rejected at the sink. Do the following to confirm and fix the error.

  1. check what setting is enabled under copy activity --> settings --> fault tolerance. if it is skip incompatiable rows, the job won't fail even if the rows are rejected.
  2. Under copy activity settings tick enable logging and output the rejected rows to your storage account.
  3. The reason for skipping the rows can be identified from this logs.
  4. Make necessary changes to the sink table and re run the job.
Hijesh V L
  • 191
  • 1
  • 5