1

I have a multi-tenant azure application where they may be many different jobs queued up. Each job consists of many individual tasks which are stored in a queue.

For both both user reasons, and technical reasons, a particular job should not consume too much of the work :

User : Don't want users to get stuck behind a huge job where they see no progress for a long time Technical : Processing the tasks involves hitting some limited resources on the network, where you can only have say 10 simultanious connections, but you may have thousands of tasks that will eventually need that connection.

To make it make it somewhat more complicated, some jobs are "related" in that they would consume the same resources, so the limit of 10 workers should be across those jobs

Currently we are using seperate queues for each job, which works ok, but we have to create them and clean them up on the fly, and have a queue of queues for the workers to find the appropriate queue

Also, this makes it difficult/impossible to do the related jobs.

Is there a good design pattern for this?

David Makogon
  • 69,407
  • 21
  • 141
  • 189
Jason Coyne
  • 6,509
  • 8
  • 40
  • 70

1 Answers1

0

Here is a design pattern you may wish to consider. It supports architecture simplicity and low cost by using a single queue and fail over safety by en-queueing all tasks.

Use a single queue for all messages. Refer to this entry (including my answers) for ideas on worker throttling and queues supporting multiple message types. How to read from multiple queues in real-world?

You have described your process as multiple jobs each having multiple tasks.

  • En-queue by adding the job (message) to queue.

  • The job message processor, performs the first task, adds the subsequent task (message) to the queue and then deletes the initial job (message). This pattern will ensure that the job/task is never lost even on a machine recycle. (Assuming and requires that all your tasks are idempotent).

  • Subsequent task processors do essentially the same thing until there are no more tasks.

  • If a task takes more time that the queue message visibility
    timeout, keep prolonging the visibility.
    Reference: Is it possible to change an Azure queue message invisibility timeout without posting the data?

Community
  • 1
  • 1
hocho
  • 1,753
  • 1
  • 11
  • 14
  • This does not address the issue of having a single job consuming all of the worker roles – Jason Coyne Jun 25 '12 at 13:07
  • Maybe I did not understand you question accurately. My answer suggests that the tasks in a job are not all placed on a queue together. Rather, each job-task on completion, places the next job-task on the queue. That way only a single job-task is ever being worked on by any of the worker roles. – hocho Jun 27 '12 at 16:57