0

I am trying to copy many containers from one Azure blob storage account to another blob storage account.

I am using the Microsoft.WindowsAzure.Storage.DataMovement library for C# and the TransferManager.CopyDirectoryAsync method.

I have no problems discovering all my containers, but when I try and copy a container, I get no feedback about blobs that have been copied. I want to copy millions of blobs so I want feedback for when this is finished. I am using an asynchronous server-side copy, I.E. isServiceCopy = true

I am calling 

return await TransferManager.CopyDirectoryAsync(sourceDirectory, 
destinationDirectory, true, options, null).ConfigureAwait(false);

and since I am awaiting I am expecting it not to pass from this line until the copy is complete. But it passes almost immediately, and if I inspect the NumberOfFilesTransferred on the TransferStatus result it is always 0. Even if I check the result repeatedly in a while loop this number never changes from 0. However, if I check in the Azure portal, all the expected files have been copied successfully.

I have also tried setting the ProgressHandler on DirectoryTransferContext, and it gets called a few times, but all the properties are 0.

Does anyone know why I am not getting any feedback from the copy?

Thanks in advance. Chris

ChrisBellew
  • 1,144
  • 2
  • 12
  • 27
  • Per my local testing, the code works well on my side. Could you share more code snippet for further troubleshooting? – Zhaoxing Lu Nov 20 '17 at 10:34
  • Hi @ZhaoxingLu-Microsoft, thanks for your response. I have uploaded my console app code to here https://gist.github.com/ChrisBellew/c4c938c6d5e2ddab4c8580503d1f890d The problem is that `CopyDirectoryAsync` on line 121 completes without the copy actually happening, and also `CountBlobsAsync` on line 97 always returns 0...presumably because the copy isn't actually working. Thanks for your help! – ChrisBellew Nov 21 '17 at 01:09

2 Answers2

1

This is incorrect:

CloudBlobDirectory sourceDirectory = sourceContainer.GetDirectoryReference(".");
CloudBlobDirectory destinationDirectory = destinationContainer.GetDirectoryReference(".");

Note that Azure Blob Storage doesn't have a real folder hierarchy, the argument here is actually a prefix of the blobs within a blob container. Therefore, please change the code to:

CloudBlobDirectory sourceDirectory = sourceContainer.GetDirectoryReference(string.Empty);
CloudBlobDirectory destinationDirectory = destinationContainer.GetDirectoryReference(string.Empty);
Zhaoxing Lu
  • 6,319
  • 18
  • 41
0

You can poll the CopyState of the blob. For details see https://stackoverflow.com/a/42255582

John Rusk - MSFT
  • 613
  • 5
  • 10
  • Thanks for your response John, but I don't think I can feasibly do this. I have to copy millions of blobs and this could take a prohibitive amount of time. Thanks anyway – ChrisBellew Nov 21 '17 at 01:11
  • Ah, that's tricky. Depending on your needs, and budget, I guess you could spin up a VM in the same region as your destination account, and run your copy code there with `isServiceCopy=false`. I.e. copy synchronously, so that you can see what's going on, but run the copy code in the cloud so you don't have to drag the data across your local network connection. – John Rusk - MSFT Nov 21 '17 at 03:08
  • Or maybe you could still run with `isServiceCopy=true` but instead of polling _repeatedly_ as in the original linked thread, you could just wait a while, and then poll each blob _once_, then wait a while (e.g. 15 mins) and poll each blob _once_ again - each time polling only those that have not completed yet. I.e. just reduce your polling frequency to something that makes it manageable. – John Rusk - MSFT Nov 21 '17 at 03:11