data factory loses permissions when copying from data lake (gen1) to blob storage

Question

Data factory gives me this error when attempting to copy from data lake gen1 to blob storage:

"message": "Failure happened on 'Sink' side. ErrorCode=UserErrorFailedFileOperation,
'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Upload file failed at
 path myblobcontainer\\file_that_im_tryin_to_copy.xml.,Source=Microsoft.DataTransfer.Common,''Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Failed to read a 'AzureDataLakeStore' file. File path: 'SourceFolderInDataLake/2019/09/26/SomeOtherFile.usql'.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Net.WebException,Message=The remote server returned an error: (403) Forbidden.

I have a U-SQL Script activity that will execute 1-Patient.usql:

The next activity is a copy data step:

Source

Sink

I have configured roles/permissions using this tutorial.

I can solve this issue by going to Data Explorer --> Access:

I then click on Advanced:

After clicking on Apply to all children, then the copying works fine!

Please note that prior to the Copy Data activity, data factory is executing usql script inside of the gen1. The script is stored in gen1, and it generates files inside of data lake as well as folders. There is never any permissions issue running this script.

What am I doing wrong?

Answer 1

I can reproduce your issue. Actually the Apply folder permissions to sub-folders is not necessary. The issue should be caused by the access control of data lake gen1, the key to the problem is the order in which files are uploaded and permissions are set.

You could check the Access control in Azure Data Lake Storage Gen1 first and refer to the information below which was based my test.

I suppose you add the permissions like below to the root /.

If your file is already existing before setting the permission, it will be affected by the operation, i.e. the access to the file will be set, you can access the file.

But if you upload the file or create a new folder after setting the permission, the folder and file will not have the access, you will not be able to access them. You could select the file, click the Access to check directly.

After setting the permission above, then if you set the A default permission entry, it will not affect the existing folders and files, but if you create new folders and files, you will get the access of all of them. i.e. the old folder and file still have not access, the new ones will have access. If you want to get the access to the old ones, just add the permissions like the screenshot again, the Apply folder permissions to sub-folders is the same logic.

So in conclusion, if you want to your service principal/MSI access all the files in your data lake, you could add the third option An access permission entry and a default permission entry, then you will be able to access both the existing and new folders/files.