Processing a Queue in Parallel

Question

I have a class which processes messages:

public abstract class ProcessingBase {
    public bool IsBusy { get; set; }
    public Queue<CustomMessage> PendingMessages { get; private set; }
    public abstract MessageProcessingResult Process();
    ...

In the processing modules I've created so far, one uses a queue to process messages, sending them over a socket and the other sends emails. This works fine for individual processing modules, but suppose I wanted to chain these two?

I was thinking I could do something like:

public class ChainProcessor : ProcessingBase {
    public List<ProcessingBase> Processors { get; set; }
    public override MessageProcessingResult Process() {
                if (IsBusy)
                    return null;
                IsBusy = true;

                CustomMessage msg = null;
                this.ProcessedMessages = new List<CustomMessage>();

                // create clone of queue 
                var messagesToSend = new Queue<CustomMessage>(this.PendingMessages);
                this.PendingMessages.Clear();
                while (messagesToSend.Count > 0 && (msg = messagesToSend.Dequeue()) != null) {
                    foreach (var processor in this.Processors) {
                        // something with yield return?
                    }
                }

It's the actual chaining which I'm shaky on. Ideally I'd want to process them in a kind of 'wave'. Eg:

Module 1 - Processed message A
Module 2 - Processed message A
Module 1 - Processed message B
Module 2 - Processed message B
Module 3 - Processed message A
Module 1 - Processed message C

Where each message is passed through the chain, with messages constantly entering and leaving as they move through. What would be the best way of doing this? Or am I limited to sequentially passing each message through the entire chain?

Ie (What I don't want): Module 1 - Processed message A Module 2 - Processed message A Module 3 - Processed message A Module 1 - Processed message B Module 2 - Processed message B Module 3 - Processed message B Module 1 - Processed message C

EDIT: I was hoping I could do something where the first processor would yield return back to the 'chain processor', which would pass that message onto module 2, then perhaps the chain could start the next message on processor1

I don't really mind what thread, what's important is that they're not processed sequentially (question updated). — Echilon, Jul 04 '12 at 12:50

IvoTops · Answer 1 · 2012-07-04T15:37:45.873

I think you can use a

   BlockingCollection<CustomMessage>

for each module. You can then create multiple threads to read from each colelction and do the module's processing work. After that put it in the next module's queue.

By creating multiple threads you can work in parallel. The BlockingCollection is threadsafe, fast and blocking. Very convenient. See http://msdn.microsoft.com/en-us/library/dd267312.aspx

score 0 · Answer 2 · answered Oct 25 '12 at 21:42

You are looking for thread safety ConcurrentQueue and AutoResetEvent, witch can wake up sleeping thread (see examples).

Each worker run on its own thread (if you have some work for main thread) and after it does its job and empties its input queue, call AutoResetEvent.WaitOne(). If input queue is filled by some item, call AutoResetEvent.Set(), which starts the process again.

You can chain these two workers by creating second queue with items processed by first worker. After first worker is done with message, adds item into output queue and call AutoResetEvent.Set().

Read and think twice, I do not know your situation, so you can optimize the process for your needs.

Good luck. ;-)

Processing a Queue in Parallel

2 Answers2