Reading Azure blob content while copying initiated through CopyBlob API is in progress

Question

I used to initiate copy blob and while copying was in progress we were able to read few bytes (e.g. first 512 bytes) of the destination blob. See below code that initiate copy blob and then read the first 512 bytes while copying is in progress.

But now blobStream.Read always fills the data buffer with zero not with the actual bytes. I have tried with latest azure storage client lib also, but output is same. But as soon as the copy finishes I can read the actual bytes.

Someone please let me know whether this is a bug introduced in the latest azure storage service and possible workaround (if any)?

CloudBlobClient client = CloudStorageAccount.Parse(
    string.Format(
        "DefaultEndpointsProtocol={0};AccountName={1};AccountKey={2}",
        "http",
        "<destination-account-name>",
        "<destination-account-key>"
    )
).CreateCloudBlobClient();


var vhdsContainer = client.GetContainerReference("<destination-container-name>");
CloudBlob destinationBlob = vhdsContainer.GetPageBlobReference("test-1-b5c1d3f4-23d5-40e4-adc7-236553f8d62d-1.vhd");
destinationBlob.StartCopyFromBlob(
    new Uri("http://<source-storage>.blob.core.windows.net/<source-container-name>/test-4-d803ca0a-5d98-4be8-8895-2a9d15ec3974-1.vhd"), null, null, null);


CloudBlob destBlob = vhdsContainer.GetBlobReference("test-1-b5c1d3f4-23d5-40e4-adc7-236553f8d62d-1.vhd");
int maxWaitTime = 3*60000;//let's wait for a maximum of 3 minute
do
{
    destBlob.FetchAttributes();
    if (destBlob.CopyState.BytesCopied > 2048 || maxWaitTime <= 0)
    {
        break;
    }
    maxWaitTime -= 1000;
    Thread.Sleep(1000);
 }
 while (true);

 var data = new byte[512];
 using (BlobStream blobStream = destBlob.OpenRead())
 {
     blobStream.Seek(0, System.IO.SeekOrigin.Begin);
     blobStream.Read(data, 0, 512);
 }

score 2 · Answer 1 · answered Oct 10 '13 at 05:12

2

As you may already know, copy blob operation is asynchronous. When you call StartCopyFromBlob, the operation is queued. There is no guarantee that the operation will start immediately. This could be the reason for not able to read meaningful data immediately after start copying operation.

One possible thing you could do is check the copy status periodically after calling the copy operation. Once you see that the copy operation has begun, then you can start reading from the destination blob.

To check if the copy operation has started, you could fetch blob attributes in a loop and check for CopyState property of the blob. Once you see some bytes are copied, you can break out of the loop. However keep in mind that you may end up looping in this loop for a long time if the copy operation does not start in a reasonable time frame.

So instead of doing:

Thread.Sleep(4000)

You could do something like:

int maxWaitTime = 10000;//let's wait for a maximum of 10 seconds
do
{
    destBlob.FetchAttributes();
    if (destBlob.CopyState.BytesCopied > 0 || maxWaitTime <= 0)
    {
        break;
    }
    maxWaitTime -= 1000;
    Thread.Sleep(1000);
}
while (true);

answered Oct 10 '13 at 05:12

Gaurav Mantri

128,066
12
206
241

Hi Gaurav, I did wait for ~30 minutes. From the CopyState.BytesCopied I can see 31209816064 bytes out of 32212255232 has been copied i.e. 96% copying is done. In this case also reading first few bytes returned zeros. But a read after copy completion returns actual bytes. Please note that this scenario was working earlier recently I see this issue. – Anu Thomas Chandy Oct 10 '13 at 18:21
That's weird. Let me also try it on my end. What version of storage client library are you using? – Gaurav Mantri Oct 10 '13 at 18:35
I have tried both 1.7 and latest version 2.1. Please note that when you use 2.1 you need to set if-match header to * so that blob service will not throw 'Pre-Condition failed' exception. Please see my answer http://stackoverflow.com/questions/5094089/azure-blob-the-condition-specified-using-http-conditional-headers-is-not-met/19278398#19278398 – Anu Thomas Chandy Oct 10 '13 at 19:36
Do you see same behavior with 1.7 & 2.1 versions of the library or does it work with 1.7 but not with 2.1? – Gaurav Mantri Oct 10 '13 at 19:53
I can see the same for both versions. – Anu Thomas Chandy Oct 10 '13 at 20:00

score 0 · Accepted Answer · answered Oct 14 '13 at 22:23

Looks like user has to wait for copy to finish to read actual bytes from the destination blob.

The CopyState::BytesCopied can be used to track the number of bytes copied, user should not assume that destination blob is ready for read based on CopyState::BytesCopied. Read only after CopyState::Status become 'Success'.

Reading Azure blob content while copying initiated through CopyBlob API is in progress

2 Answers2