Continuous data stream using DataFlow and RX stops processing

Question

I have setup a system to read messages from a queue indefinitely, then farm them off using Rx and TPL DataFlow.

For some reason, after a few hundred messages the ActionBlock stops running hangs and I cannot figure out why. this.GetMessages() continues to fire but this.ProcessMessages no longer does.

var source = Observable
    .Timer(TimeSpan.FromMilliseconds(1), TimeSpan.FromMilliseconds(1))
    .SelectMany(x => this.GetMessages());

var actionBlock = new ActionBlock<List<QueueStream>>(
    this.ProcessMessages,
    new ExecutionDataflowBlockOptions
    {
        MaxDegreeOfParallelism = Environment.ProcessorCount * 2,
    });

using (source.Subscribe(actionBlock.AsObserver()))
{
    while (this.Run)
    {
        await Task.Delay(TimeSpan.FromSeconds(1));
    }
}

actionBlock.Complete();
await actionBlock.Completion;

The reader - Note that this actually continues to run

private async Task<List<QueueStream>> GetMessages()
{
    var messageList = new List<QueueStream>();
    var taskList = new List<Task>();

    // Add up to N items in our queue
    for (var i = 0; i < 25; i++)
    {
        var task = this
            .ReadAndParseQueue()
            .ContinueWith(async queueStreamTask =>
                {
                    var queueStream = await queueStreamTask;
                    if (queueStream != null)
                    {
                        messageList.Add(queueStream);
                    }
                });

        taskList.Add(task);
    }

    await Task.WhenAll(taskList);

    return messageList;
}

The writer - after a few hundred messages this stops getting hit

private async Task ProcessMessages(List<QueueStream> streams)
{
    var tasks = new List<Task>();
    foreach (var queueStream in streams)
    {
        tasks.Add(this.ProcessMessage(queueStream));
    }

    await Task.WhenAll(tasks);
}

You really need to provide a [mcve]. Something that we can copy-paste-and-run code that reproduces this problem. — Enigmativity, Apr 20 '17 at 15:01
@Chris what is the state of the `ActionBlock` once it _hangs_, it is likely that it is faulted. Any exception thrown from within `ProcessMessages` will not be observed until you `await` completion which in your case would never happen if `this.run` is always set. — JSteward, Apr 20 '17 at 17:13

score 1 · Answer 1 · answered Apr 20 '17 at 19:51

Are you sure that your source keeps running in this case? There is an infinite loop in your code, but, if error occurs or this.Run is unset, it will stop, and after that you have these lines:

actionBlock.Complete();
await actionBlock.Completion;

which actually prevents the actionBlock to accept new mesasges, so ProcessMessages will never being called, as messages are simply being ignored.

Continuous data stream using DataFlow and RX stops processing

1 Answers1