Delay seeing Azure queue message

Question

I am running my cloud service in debug mode in Visual Studio and I'm seeing a large delay from the button click event on my live site and hitting the breakpoint in my debug session of the cloud service. It takes about 20-30 seconds from button click until the queue message is present. Is this normal? Is this due to running in debug mode, or does this roughly represent what it would look like in production? This is my first cloud service project, so I'm still learning the knobs to turn. I am the only person hitting the cloud service as this is still in development.

EDIT to add code calls.

Here are the calls I am making from the website. The queue message is just a filename, so the message payload is very small.

CloudQueueClient queueClient = storageAccount.CreateCloudQueueClient();

//Retrieve a reference to a queue.
CloudQueue queue = queueClient.GetQueueReference("queue");

//Create the queue if it doesn't already exist.
queue.CreateIfNotExists();

//Create a message and add it to the queue.
CloudQueueMessage message = new CloudQueueMessage(filename);

queue.AddMessage(message, timeToLive: TimeSpan.FromMinutes(1), initialVisibilityDelay: null);

This call takes around 30 seconds to be picked up here in the WorkerRole:

        // Retrieve a reference to a container.
        CloudBlobContainer container = blobClient.GetContainerReference("mycontainer");
        CloudBlobContainer hmtlcontainer = blobClient.GetContainerReference("conthtml");

        // Create the container if it doesn't already exist.
        container.CreateIfNotExists();
        hmtlcontainer.CreateIfNotExists();

        // Create the queue client
        CloudQueueClient queueClient = storageAccount.CreateCloudQueueClient();

        // Retrieve a reference to a queue
        CloudQueue queue = queueClient.GetQueueReference("queue");

        try
        {
            // Look for a new queue message
            CloudQueueMessage peekedMessage = queue.GetMessage(visibilityTimeout: TimeSpan.FromSeconds(3)); 

Answer 1

No, the latency should not be that long at all. I would definitely think it has more to do with remote debugging. There is a lot of information that goes between the IDE and a debugged process. When remote debugging like that I can see it taking a while. I tend to use the emulator for my debugging unless absolutely necessary to debug from a running cloud environment.

That being said, if I were you I'd add some telemetry to your logging to see when you picked up messages, how long they took to process, etc. Then when you run outside of debug you'll get a better idea of the actual processing. Also note that you can take a look at the properties of the queue message for the InsertionTime and compare that to the time the processor picks it up to see how long it sat in the queue.

Answer 2

MikeWo's answer is correct that to be sure of why you're seeing so much latency you'd need to figure out where that latency is actually coming from in your code. If you post the particular call or calls that are slow we can probably provide better advice. As it is, we're largely just speculating. That being said, I wanted to add some speculations assuming the latency has to do with Azure storage:

1. It would not explain all of the latency, but check where in the world you're actually calling to. If your storage account is in Asia and you're in California, you're going to see much more latency then if your code is running in a location closer to the data center.
2. Specific to queue messages, check that your visibility timeout is low (or 0) when you're putting the message. If your message visibility timeout is 30 seconds, it won't actually show up on gets for 30 seconds.
3. Confirm how many storage calls you're actually making. If you're creating the queue, putting messages, polling... etc, it's going to take a lot more than the quoted 10 ms for small messages. If you figure out which is slow using Mike's advice, that will help you debug further.