Blocking collections + Multiple Worker threads per blocking collection + Wait For Work Completion

Question

I have to do action in batch of 1000 message say Action A, B, C. I can do these actions in parallel. I created groups for them. To increase parallelism, I created subgroups with in each group. Task with in a subgroup needs to be executed serially. But two subgroups can execute in parallel. After a batch of 1000 finishes, I have to do some processing ie save in db. But I am unable to understand , how to wait for all the task to finish (I am not interested in waiting in middle just at the end of 1000 taks). Any suggestions are welcome.

public class OrderlyThreadPool<t> : IDisposable
{
    BlockingCollection<t> _workingqueue = null;
    Action<t> _handler = null;
    public OrderlyThreadPool(int wrkerCount, Action<t> handler)
    {
        _workingqueue = new BlockingCollection<t>();
        _handler = handler;
        Worker worker = new Worker(wrkerCount, Process); //WorkerCount is always 1
        worker.Start();
    }

    public void AddItem(t item)
    {
        _workingqueue.Add(item);
    }
    private void Process()
    {
        foreach (t item in _workingqueue.GetConsumingEnumerable())
        {
            _handler(item);
        }
    }
    public void Dispose()
    {
        _workingqueue.CompleteAdding();
        _workingqueue = null;
    }

}

public class Worker
{
    int _wrkerCount = 0;
    Action _action = null;
    public Worker(int workerCount, Action action)
    {
        _wrkerCount = workerCount;
        _action = action;

    }
    public void Start()
    {
        // Create and start a separate Task for each consumer:
        for (int i = 0; i < _wrkerCount; i++)
        {
            Task.Factory.StartNew(_action);
        }
    }
}

So basically I will create OrderlyThreadPool for each subgroup. I am recv messages from say source, which blocks if no message is available. So my code, looks like

while(true)
{

    var message = GetMsg();

    foreach(OrderlyThreadPool<Msg> a in myList)
    {
        a.AddMsg(message);

    }

    if(msgCount > 1000)
    {
    Wait for all threads to finish work;

    }
    else
    {
    msgCount =msgCount+1;

    }

}

i3arnon · Answer 1 · 2014-09-14T19:53:03.967

You start your tasks but you don't keep a reference. Simply store these tasks, expose them through the Worker and OrderlyThreadPool and use Task.WhenAll to wait for all of them to complete:

public class Worker
{
    //...
    List<Task> _tasks = new List<Task>();
    public Task Completion { get { return Task.WhenAll(_tasks); } }
    public void Start()
    {
        // Create and start a separate Task for each consumer:
        for (int i = 0; i < _wrkerCount; i++)
        {
            Tasks.Add(Task.Factory.StartNew(_action));
        }
    }
}

public class OrderlyThreadPool<t> : IDisposable
{
    //...
    public Task Completion { get { return _worker.Completion; }}
}

await Task.WhenAll(myList.Select(orderlyThreadPool => orderlyThreadPool.Completion));

However, you should probably consider using TPL Dataflow instead. It's an actor-based framework that encapsulates completion, batching, concurrency levels and so forth...

Hello, My sitchuation, say I have just 6 members in myList.So for 1 message 6 task will be executing. I want to know when all the task have executed 1000 times. ie 1000 messages have been processed. So I dont know how can I do that, any suggestion — Rajesh Gaur, Sep 14 '14 at 19:58

Blocking collections + Multiple Worker threads per blocking collection + Wait For Work Completion

1 Answers1