I want to implement the consumer/producer pattern using the BufferBlock<T>
, in the context of an ASP.NET request, but I also need:
- to buffer the 'produced' messages to prevent the queue filling up the available memory (say 50 items in the buffer at a time)
- The producer will 'produce' continuously, and the thread it runs on will be shutdown shortly after it produces a bunch of messages. (as the ASP.NET request ends)
- The consumer will 'consume' continuously, as multiple producers (ASP.NET requests) produce a bunch of messages each, so we think having a separate thread doing the consuming is the way to go.
[The component I ultimately need intermediates logging calls within a long running ASP.NET hosted REST service, where each request 'produces' many logging messages as it executes, and the consumer eventually 'consumes' them by writing them to some persistence store (which is comparatively slower, over the network).]
We've been following Steven Cleary's guidance on using the BufferBlock<T>
here, but it seems that all these examples call the BufferBlock<T>.Complete()
method which ends the production of messages. (i.e. once Complete()
is called, calling SendAsync()
is ignored by the block.)
So how can we use the BufferBlock<T>
in a 'continuous' mode where the consumer continuously consumes while the producer continuously produces? I think without calling Complete()
.
We are using BoundedCapacity
to define the buffer:
var _queue = new BufferBlock<TMessage>(new DataflowBlockOptions
{
BoundedCapacity = 50
});
and then later...to start the consumer:
Task.Run(async () =>
{
while (true)
{
try
{
while (await _queue.OutputAvailableAsync())
{
var message = await _queue.ReceiveAsync();
try
{
_storage.Store(message)
}
catch (Exception)
{
//Log and Ignore exception and continue
}
}
}
catch (Exception)
{
//Log and Ignore exception and continue
}
}
});
So, with the consumer running endlessly, we now want to continuously call SendAsync()
(by each ASP.NET request thread) and have messages queue up, while the consumer (an a thread pool thread) continuously consumes them.
How do you use BufferBlock<T>
or other TPL types to achieve this?
UPDATE (22/7/2016):
I ended up linking an ActionBlock to the BufferBlock and just passing the delegate _storage.Store(message)
to the ActionBlock for easier maintenance, and to avoid the whole while loop and Complete()
thing.