4

I am getting the following error in a pipeline that has Copy activity with Rest API as source and Azure Data Lake Storage Gen 2 as Sink.

"message": "Failure happened on 'Sink' side. ErrorCode=AdlsGen2OperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=ADLS Gen2 operation failed for: Operation returned an invalid status code 'Conflict'. Account: '{Storage Account Name}'. FileSystem: '{Container Name}'. Path: 'foodics_v2/Burgerizzr/transactional/_567a2g7a/2018-02-09/raw/inventory-transactions.json'. ErrorCode: 'LeaseAlreadyPresent'. Message: 'There is already a lease present.'. RequestId: 'd27f1a3d-d01f-0003-28fb-400303000000'..,Source=Microsoft.DataTransfer.ClientLibrary,''Type=Microsoft.Azure.Storage.Data.Models.ErrorSchemaException,Message=Operation returned an invalid status code 'Conflict',Source=Microsoft.DataTransfer.ClientLibrary,'",

The pipeline runs in a for loop with Batch size = 5. When I make it sequential, the error goes away, but I need to run it in parallel.

Prateek Gupta
  • 2,422
  • 2
  • 16
  • 30
sameet90
  • 123
  • 7
  • Hi @sameet90, did you get any solution for this ? – Ritesh Apr 29 '21 at 13:14
  • Hi @Ritesh, when we run it in parallel, variables are not thread safe, this may cause some problems. Can you show us more details and post an issues. – Joseph Xu May 10 '21 at 10:00
  • 1
    @JosephXu I am getting the same issue as mentioned here with the solution you provided on the post https://stackoverflow.com/questions/67298387/source-to-sink-folder-mapping-in-adf-v2 – Ritesh May 10 '21 at 12:27

1 Answers1

0

This is known issue with adf limitation variable thread parallel running. You probably trying to rename filename using variable. Your option is to run another child looping after each variable execution. i.e. variable -> Execute Pipeline

enter image description here

or remove those variable, hard coded those variable expression in azure activity. enter image description here Hope this helps

arif hassan
  • 1
  • 1