0

I want to do a certain task periodically (per day) on our Web/Worker role. I have multiple instances in my Cloud Service, and I want only one of these instances to do this task per day (for example Instance0 can do it one day, next day it could be Instance1 doing the work, but 0 and 1 will not try to do the same work during the same day/period)

Azure queues seem to be a great way to achieve this because by design only one instance will dequeue the message (assuming it deletes it after doing the work).

What I am having trouble with is figuring out a way to put only one copy of this message in the queue per day. The only way I have figured to do this is by enqueuing a message every day from Azure scheduler jobs.

My problem with Azure scheduler is the fact that I need to create a job for every single storage account I have across all of my deployments.

Is there a way to do this from within the cloud service, without taking the scheduler dependency?

Sushant
  • 1,074
  • 2
  • 13
  • 21

2 Answers2

1

If you don't want to have Scheduler dependency, consider using Blob leases as a semaphore of sorts. http://justazure.com/azure-blob-storage-part-8-blob-leases/

At a certain time of the day, have your worker instances compete to get a lease on some central storage blob. Whoever gets the lease, prevents other instances from getting that lease and can queue up messages into the queue.

Having said that, why are you afraid of Scheduler dependency? Have it kick off a single job that will queue up a "start work" message. Have your instances monitor that queue. Whoever picks up that message, can then run thru all of your storage accounts and queue up individual storage-work messages for all instances to pickup.

Igorek
  • 15,716
  • 3
  • 54
  • 92
  • As I said, I have a bunch of deployments all across the works, and in my design I have a storage account per data center, therefore there are too many of these Scheduler jobs to create and maintain (one per storage account) – Sushant Jul 10 '15 at 20:56
  • With the central lease, now I have to integrate something like Quartz.Net to schedule a periodic job to queue an item. If I use Quartz, I can use the blob lease mechanism you suggested to directly perform my task (instead of using queues) – Sushant Jul 10 '15 at 20:58
  • Maybe we're not understanding each other. Why do you need a single scheduler job per account? Have a single job, whoever processes that job creates queue messages everywhere. – Igorek Jul 10 '15 at 20:58
  • AFAIK, with azure scheduler, at the time of creating a job, we specify which storage account to enqueue the message to. The other option would be invoke an API from a scheduler job which can enqueue things everywhere, but then I have to build auth into the scheduler job. – Sushant Jul 10 '15 at 21:06
  • 1
    well, you'll have to do something somewhere. I'd recommend having scheduler kickoff API call that will generate messages. You can put basic auth into the URL w/o getting complex – Igorek Jul 10 '15 at 21:08
  • You have given me another idea,I can also queue a message to indicate that whoever processes that message should enqueue a message to all storage accounts. Then I save myself from building any kind of auth. – Sushant Jul 10 '15 at 21:10
  • 1
    uhm, that's exactly what I said in my original response (2nd part) :) – Igorek Jul 10 '15 at 21:11
0

If you know you need to perform one job a day I don't understand why you need to use a queue - you could just have a scheduled job run once per day. If the job only needs to run if a certain condition is met I would just build this logic into your scheduled task - maybe by setting a property in a Table or something like that.

Jason Hogg - MSFT
  • 1,369
  • 9
  • 10
  • how are you gonna ensure only one instances processes that job? You would have to take a blob lease at the minimum. – Sushant Jul 13 '15 at 18:16