Producer/Consumer pattern with a batched producer

Question

I'm attempting to implement a fairly simple Producer/Consumer style application with multiple producers and one consumer.

Research has led me onto the BlockingCollection<T> which is useful and allowed me to implement a long-running consumer task as below:

var c1 = Task.Factory.StartNew(() =>
{
    var buffer = new List<int>(BATCH_BUFFER_SIZE);

    foreach (var value in blockingCollection.GetConsumingEnumerable())
    {
        buffer.Add(value);
        if (buffer.Count == BATCH_BUFFER_SIZE)
        {
            ProcessItems(buffer);
            buffer.Clear();
        }
    }
});

The ProcessItems function submits the buffer to a database and it does work in batches. This solution is sub optimal however. In baron production periods it could be some time until the buffer is filled meaning the database is out of date.

A more ideal solution would be either to run the task on a 30 second timer or short circuit the foreach with a timeout.

I ran with the timer idea and came up with this:

syncTimer = new Timer(new TimerCallback(TimerElapsed), blockingCollection, 5000, 5000);

private static void TimerElapsed(object state)
{
    var buffer = new List<int>();
    var collection = ((BlockingCollection<int>)state).GetConsumingEnumerable();

    foreach (var value in collection)
    {
        buffer.Add(value);
    }

    ProcessItems(buffer);
    buffer.Clear();
}

This has the clear problem that the foreach will be blocked until the end, defeating the purpose of the timer.

Can anyone offer a direction to take? I essentially need to snapshot the BlockingCollection periodically and process the contents the clear it. Perhaps a BlockingCollection is the wrong type?

Answer 1

Instead of using GetConsumingEnumerable in the timer callback, use one of these methods instead, adding the results into a list until it returns false or you've reached a satisfactory batch size.

BlockingCollection.TryTake Method (T) - probably what you need, you don't want to perform further waiting at all.

BlockingCollection.TryTake Method (T, Int32)

BlockingCollection.TryTake Method (T, TimeSpan)

You can easily extract that out into an extension (untested):

public static IList<T> Flush<T>
(this BlockingCollection<T> collection, int maxSize = int.MaxValue)
{
     // Argument checking.

     T next;
     var result = new List<T>();

     while(result.Count < maxSize && collection.TryTake(out next))
     {
         result.Add(next);
     }

     return result;
}