0

I have a pipeline that started going very slow after a change. The issue was due to rows being skipped by the copy due to an error.

i was able to undo the change to a column formula and the pipeline runs fine now. the issue is that there was no indication of what was causing the issue.

Is there a way to get a detailed error message on why ADF is skipping rows?

Shappireritz
  • 17
  • 3
wilson_smyth
  • 1,202
  • 1
  • 14
  • 39

1 Answers1

3

  1. Using Set Variable activity, we can capture the detailed error message of previous activity when error occurred.

    • Add output of copy activity on failure to Set Variable.

    enter image description here

    • Create a variable on pipeline level to store the message.

    enter image description here

    • When you select Set variable activity, in variables drop down, it gives the list of variables available. Select the variable created to store error message.

    enter image description here

    • Instead of hardcoding the message in Value, we will capture from previous activity output. For that we can add dynamic content for the value.

    • Add the expression @activity('Copy data1').Error.Message in the dynamic content. Error.Message is used to capture error message.

    enter image description here

    enter image description here

    • Run the pipeline and you can see the Set variable activity is triggered when copy activity is failed.

    • Click on the output of set variable and you can see the detailed error message of the error occurred.

    enter image description here

  2. You can log the skipped rows or skipped files by enabling Session log in copy activity and store these log files in a Storage location.

    When you select "Warning" as logging level, it will log skipped files and skipped rows only.

    Once the activity is ran, log file is available in the linked path for download.

enter image description here

Reference : Session log in copy activity

NiharikaMoola-MT
  • 4,700
  • 1
  • 3
  • 15