0

I want to make a long-term process handler and use for it NServiceBus. The role of NServiceBus is to hold an operations of that process (some kind of batch process) The problem is that I have more than one type of long-term processes and each of them must run parallel, so pushing all messages in one queue is not that I have to do, I think.

Logic is: 1) Receive an order of a long-term process, 2) Divide it into N operations, 3) Each operation "pack" into the message and push in the queue, 4) According to the type of message, particular handler will handle messages and do the operation it holds.

I can't put all of the operations in one queue because my application should handle another messages, that requires fast response. If queue would be full of operations, another messages would wait a lot of time to be processed

So, does anyone know how to solve that problem ?

Tim Gim
  • 174
  • 1
  • 1
  • 10
  • Have you seen sagas? They are designed for long-running processes. – Udi Dahan Nov 23 '12 at 12:43
  • In addition to Sagas, it sounds like you need to draw some boundaries around the operations and consider creating separate endpoints for them. Coordination between them can be handled via Sagas. – Adam Fyles Nov 25 '12 at 14:55
  • @UdiDahan, my long-running process divides into 2+ billions operations, which may partially fail (e.g. ++++--++-++---++-). And I need to restart failed (which thrown an exceptions) operations. And queues (not sagas) is the best solution here, I think. – Tim Gim Nov 26 '12 at 05:16
  • 1
    Then I think you need to look at a more domain-specific design rather than generically leaning on either queues or sagas. These are the boundaries that Adam Fyles was talking about in his comment. – Udi Dahan Nov 26 '12 at 10:36
  • @AdamFyles all operations are the same. For example, someone asked my application to run long-therm process. He sent "OperationId" (for notification when it completed) and collection of some IDs and. Each IDs is a basis of operation. Then I divide all process to N operations like "DoOperation(1,1)", "DoOperation(2,1)", "DoOperation(3,1)". Each operation has the same logic, only Id is different. But some of operations may fail due to some technical exceptions. And I need to repeat them if failed.I already made a such service without NServiceBus or something.But I want to use all power of queues – Tim Gim Nov 27 '12 at 06:11
  • 1
    You may want to checkout putting those operations into a pipeline using ISpecifyMessageOrderHandling interface. Then you may be into the Distributor(master/workers) to load balance it all. – Adam Fyles Nov 27 '12 at 16:27

1 Answers1

0

You should properly set the number of worker threads in the access queue config settings of the long-running process endpoint.

if you are using MSMQ check out this and especially the tag <MsmqTransportConfig ErrorQueue="error" NumberOfWorkerThreads="1" MaxRetries="5"/>

Every idle worker thread pull out a message from the queue although another thread is still processing another message. In this way you shoud achieve the parallel computation requirement you described in your scenario.

Riccardo
  • 1,490
  • 2
  • 12
  • 22
  • If I will put all messages in one queue, there may be billions of messages of different types (handling all of them can take 1-2 days), while some types requires immediate handling. So, I need more than one queue... – Tim Gim Nov 24 '12 at 18:03
  • I think you are looking for a load balancer as the NServiceBus distributor(http://nservicebus.com/DistributorV3.aspx) – Riccardo Nov 26 '12 at 10:18