1

I am trying to implement job queue with MSMQ to save up some time on me implementing it in SQL. After reading around I realized MSMQ might not offer what I am after. Could you please advice me if my plan is realistic using MSMQ or recommend an alternative ?

I have number of processes picking up jobs from a queue (I might need to scale out in the future), once job is picked up processing follows, during this time job is locked to other processes by status, if needed job is chucked back (status changes again) to the queue for further processing, but physically the job still sits in the queue until completed.

MSMQ doesn't let me to keep the message in the queue while working on it, eg I can peek or read. Read takes message out of queue and peek doesn't allow changing the message (status).

Thank you

Vojtiik
  • 2,558
  • 1
  • 17
  • 21

2 Answers2

1

Using MSMQ as a datastore is probably bad as it's not designed for storage at all. Unless the queues are transactional the messages may not even get written to disk.

Certainly updating queue items in-situ is not supported for the reasons you state.

If you don't want a full blown relational DB you could use an in-memory cache of some kind, like memcached, or a cheap object db like raven.

tom redfern
  • 30,562
  • 14
  • 91
  • 126
0

Take a look at RabbitMQ, or many of the other messages queues. Most offer this functionality out of the box.

For example. RabbitMQ calls what you are describing, Work Queues. Multiple consumers can pull from the same queue and not pull the same item. Furthermore, if you use acknowledgements and the processing fails, the item is not removed from the queue.

.net examples: https://www.rabbitmq.com/tutorials/tutorial-two-dotnet.html

EDIT: After using MSMQ myself, it would probably work very well for what you are doing, as far as I can tell. The key is to use transactions and multiple queues. For example, each status should have it's own queue. It's fairly safe to "move" messages from one queue to another since it occurs within a transaction. This moving of messages is essentially your change of status.

We also use the Message Extension byte array for storing message metadata, like status. This way we don't have to alter the actual message when moving it to another queue.

MSMQ and queues in general, require a different set of patterns than what most programmers are use to. Keep that in mind.

Perhaps, if you can give more information on why you need to peek for messages that are currently in process, there would be a way to handle that scenario with MSMQ. You could always add a database for additional tracking.

Michael Silver
  • 483
  • 5
  • 15