5

I was wondering about the best practices in synchronizing multiple azure instances that run the same role. More precisely, I want to prevent several worker roles to work on the same work-unit.

Azure queues do not seem to help on this matter. One option is to use an sql table with locks and stored procedures; but using sql synchronization in Azure seems a bit awkward.

Any ideas?

Edit, my detailed(but simplified problem) is as follows:

  • There are n targets.
  • A unit of work must be done on each target at a specified interval (say 30 seconds - but it is different for each target).
  • I have m workers (hosted in h instances).
  • Processing a unit of work could take anything between 10 seconds and 1 hour.

The idea is that I have a scheduler that puts units of work in an Azure queue, and each of the m workers will read these and process them.

The problem:

  • worker1 starts working on unit1 (which is regarding target1) - this one will take long, say 10 minutes
  • 30 seconds pass
  • the scheduler puts another unit of work for target1, say unit13
  • worker2 starts working on unit13, against the same target1 - not good

I have some ideas, but they don't seem cloudy enough, so I am interested to see what solutions would you apply for this problem.

David Makogon
  • 69,407
  • 21
  • 141
  • 189
Flavius
  • 111
  • 1
  • 12
  • 2
    Why do you think queues won't work here? Queues are the traditional way to coordinate work that needs to be done once. There are some nuances sure, but the 90% case is with queues. – dunnry Aug 31 '11 at 16:22
  • I agree with David's answer, queues are generally a good option. Although there are times when you can't do queues. But if that's the case, describe your problem in detail, and we'll try to provide a better answer. – Anže Vodovnik Sep 01 '11 at 07:04
  • In the mean time, I've posted an idea to UserVoice for Azure - http://entlib.uservoice.com/forums/101257-windows-azure-integration-pack/suggestions/2050987-distributed-synchronization?ref=title something that might be useful for those cases when Queues just won't work – Anže Vodovnik Sep 01 '11 at 07:07

3 Answers3

4

I've just written a couple blog posts about using blob leases to do this sort of thing. See http://blog.smarx.com/posts/managing-concurrency-in-windows-azure-with-leases and http://blog.smarx.com/posts/building-a-task-scheduler-in-windows-azure.

user94559
  • 59,196
  • 6
  • 103
  • 103
  • Thank you very much for the answer. Actually, my original idea was to use queues for task distribution and blobs for synchronization, but I was using a different library. I will also take yours and test it. – Flavius Sep 09 '11 at 10:25
2

dunnry is spot-on: queues work great for preventing multiple instances from working on the same work item. When you call GetMessage, the message you retrieve is now invisible for the timespan you specify (default: 30 seconds). In that timespan, no other reader can retrieve this queue message.

Having said that: You need to ensure your processing is idempotent. In the case where your processing takes longer than the invisibility timespan, the message becomes visible again. At this point, the original reader cannot delete the message, and someo other reader can read the message (making it once again invisible). In this case, it's possible that you re-process the same message. You'll need to carefully set your timeout window to avoid this as a general rule.

Note: Each CloudQueueMessage has aDequeueCount property, so you can determine if the message has been seen more than once (and so you can also deal with poison messages).

David Makogon
  • 69,407
  • 21
  • 141
  • 189
  • One minor clarification - the owner of the last pop receipt is the only one that can delete the queue message. So, even in the case that a message went 'visible' again, it could still be deleted by the original reader as long as it was not subsequently de-queued in the interim (which would generate a new pop receipt). – dunnry Aug 31 '11 at 20:34
0

CloudFX has a PrimaryInstanceManager class that can be used for some of these scenarios.

Chriseyre2000
  • 2,053
  • 1
  • 15
  • 28