How to stop processing pipeline on faulty block?

Question

How can I stop processing of DataFlow blocks if one of the blocks made decision that an error occurred, preventing next blocks to run. I thought a block can throw an exception, but not sure what is the proper way to stop further processing pipeline.

UPDATE:

private async void buttonDataFlow_Click(object sender, EventArgs e)
{
    var cells = objectListView.CheckedObjects.Cast<Cell>().ToList();
    if (cells == null)
        return;

    var blockPrepare = new TransformBlock<Cell, Cell>(new Func<Cell, Task<Cell>>(Prepare),
    new ExecutionDataflowBlockOptions
    {
        BoundedCapacity = 10000,
        MaxDegreeOfParallelism = Environment.ProcessorCount,
    });

    var blockPreparationFeedback = new TransformBlock<Cell, Cell>(new Func<Cell, Task<Cell>>(PreparationFeedback),
    new ExecutionDataflowBlockOptions
    {
        BoundedCapacity = 10000,
        MaxDegreeOfParallelism = Environment.ProcessorCount,
    });

    var blockTestMover = new ActionBlock<Cell>(new Func<Cell, Task>(TestMover),
    new ExecutionDataflowBlockOptions
    {
        BoundedCapacity = 10000,
        MaxDegreeOfParallelism = Environment.ProcessorCount,
    });

    blockPrepare.LinkTo(blockPreparationFeedback, new DataflowLinkOptions { PropagateCompletion = true });
    blockPreparationFeedback.LinkTo(blockTestMover, new DataflowLinkOptions { PropagateCompletion = true });

    foreach (Cell c in cells)
    {
        var progressHandler = new Progress<string>(value =>
        {
            c.Status = value;
        });

        c.Progress = progressHandler as IProgress<string>;
        blockPrepare.Post(c);
    };

    blockPrepare.Complete();
    try
    {
        await blockTestMover.Completion;
    }
    catch(Exception ee)
    {
        Console.WriteLine(ee.Message);
    }

    Console.WriteLine("Done");
}

UPDATE 2:

    public ITargetBlock<TInput> CreateExceptionCatchingActionBlock<TInput>(
                    Func<TInput, Task> action,
                    Action<Exception> exceptionHandler,
                    ExecutionDataflowBlockOptions dataflowBlockOptions)
    {
        return new ActionBlock<TInput>(async input =>
        {
            try
            {
                await action(input);
            }
            catch (Exception ex)
            {
                exceptionHandler(ex);
            }
        }, dataflowBlockOptions);
    }

You could take a look as Stephen Cleary's minimalistic [`Try`](https://github.com/StephenCleary/Try) library. It allow to pass a message through all the blocks of a pipeline, and then observe any exception that occurred to this message at the end. — Theodor Zoulias, Jun 26 '20 at 16:59

score 4 · Accepted Answer · answered Jul 17 '16 at 17:18

If what you want is that an exception in a block means the current items does go further in the pipeline, but processing of other items should continue without interruption, then you can do that by creating a block that produces one item if processing succeeds, but produces zero items when an exception is thrown:

public IPropagatorBlock<TInput, TOutput> CreateExceptionCatchingTransformBlock<TInput, TOutput>(
    Func<TInput, Task<TOutput>> transform,
    Action<Exception> exceptionHandler,
    ExecutionDataflowBlockOptions dataflowBlockOptions)
{
    return new TransformManyBlock<TInput, TOutput>(async input =>
    {
        try
        {
            var result = await transform(input);
            return new[] { result };
        }
        catch (Exception ex)
        {
            exceptionHandler(ex);

            return Enumerable.Empty<TOutput>();
        }
    }, dataflowBlockOptions);
}

just one last thing.. Although ActionBlock is last block, I've tried to make similar method to create for it. My post is updated with an attempt to create such method. But compiler complains that cannot await 'void'. I can understand him, but not sure how to solve it. — Pablo, Jul 17 '16 at 20:27

score 2 · Answer 2 · answered Jul 17 '16 at 13:18

2

If you have a pipeline, you're probably already using PropagateCompletion = true. That means if one block in the pipeline fails with an exception, all the blocks after it are going to fail as well.

What remains is to stop all the blocks that are before the block that failed. To do that, you can wait for the Completion of the last block in the pipeline. If doing that throws, fail the first block by calling Fault() on it. The code could look like this:

// set up your pipeline

try
{
    await lastBlock.Completion;
}
catch (Exception ex)
{
    ((IDataflowBlock)firstBlock).Fault(ex);

    throw; // or whatever is appropriate to propagate the exception up
}

answered Jul 17 '16 at 13:18

svick

236,525
50
385
514

This is cancelling everything, even those jobs, that are not faulty. I mean I am posting number of jobs to pipeline in foreach loop and all of them got cancelled if one of them is faulty. First 8 items(8 because I have 8 core CPU) are running, one of them is not going further and rest are completing. But apart from that 8 items, others are no more processed. If no exception is occurring, then all items are processed. I've updated source code in my post. – Pablo Jul 17 '16 at 13:38
@Pablo I thought that's what you wanted, "to stop further processing [in the] pipeline". – svick Jul 17 '16 at 15:55
I thought pipeline is when I post to block a job with some input data, then the job is passing to another block and so on, according to linked blocks. If I post only once, then the chain of the blocks are getting canceled as expected. But if I post multiple times, then all "posts" are getting cancelled, even those, which are running without any exception. In other words if I am posting input data [1, 2, 3, 4] to chain of blocks and if only when I post 2 brings to exception, then 1, 3, 4 to finish the job as expected. I probably used wrong terms, but I am not sure how to describe better. – Pablo Jul 17 '16 at 16:30

How to stop processing pipeline on faulty block?

2 Answers2