TPL Dataflow execution output order

Question

In the below test code, I am expecting this outcome:

1, 2000
2, 4000
3, 6000

However the actual outcome is:

3, 6000    
2, 4000
1, 2000

Moreover, i only see the outcome on screen after 6 seconds. Which means anything input that is competed is waiting and processed to the next stage.

How can i make the pipeline to spit out the result per input a soon as is completed?

    public static void Run()
    {
        Func<int, string> fn = n =>
        {
            var sleep = n * 2000;
            Thread.Sleep(sleep);
            return n + ", " + sleep;
        };

        var opts = new ExecutionDataflowBlockOptions
        {
            MaxDegreeOfParallelism = 4
        };

        var transformBlock = new TransformBlock<int, string>(fn, opts);
        var bufferBlock = new BufferBlock<string>(opts);

        transformBlock.LinkTo(bufferBlock, new DataflowLinkOptions { PropagateCompletion = true });

        for (var i = 3; i > 0; i--)
            transformBlock.Post(i);

        Console.WriteLine(bufferBlock.Receive());
        Console.WriteLine(bufferBlock.Receive());
        Console.WriteLine(bufferBlock.Receive());
    }

That's not the *output* order. That's the execution order of four concurrent tasks. ActionBlock has no output. If you used *transformblock* on the other hand that linked eg to a BufferBlock, you'd see that the results come out in order — Panagiotis Kanavos, Oct 10 '18 at 07:50
BTW you are forcibly putting *all* TransformBlock tasts to sleep for up to 6 seconds. With only 3 messages starting from the one with the largest delay, a wait of 6 seconds is guaranteed — Panagiotis Kanavos, Oct 10 '18 at 07:52
Finally, the input is `3, 2, 1`. That's what `for (var i = 3; i > 0; i--)` produces. There's no reason to expect `1,2,3` as the result. What do you really want to do? Ensure the order, or are you OK with returning results out of order? — Panagiotis Kanavos, Oct 10 '18 at 07:54
i updated source code to use buffer, and still getting same results. also i am not getting how am i putting all the TransformBlock to sleep for 6 sec. they are supposed to be running in parallel — user1912383, Oct 10 '18 at 08:02
There's nothing strange about the results. They *are* running in parallel, the order *is* preserved. You are posting the messages in reverse, so you get results in reverse. If they *didn't* run in parallel you'd get 9 seconds of delay. Right now, in order to preserve order the first result has to match the first message which is forced to wait for 6 seconds. By that time all other results are ready and waiting their turn — Panagiotis Kanavos, Oct 10 '18 at 08:41

Haukinger · Answer 1 · 2018-10-10T08:19:28.473

5

By default, data flow preserves the order of messages even if they are processed in parallel.

To transform the messages as fast as possible, i.e. out of order, set EnsureOrdered to false in the options of your TransformBlock.

Be sure to use an up to date version of dataflow (currently the nuget package System.Threading.Tasks.Dataflow exists in version 4.9), as EnsureOrdered has not always been around.

Example:

class Program
{
    static void Main( string[] args )
    {
        var transformBlock = new TransformBlock<int, int>( x =>
        {
            Thread.Sleep( x );
            return x;
        }, new ExecutionDataflowBlockOptions {EnsureOrdered = false, MaxDegreeOfParallelism = 10} );
        var actionBlock = new ActionBlock<int>( x => Console.WriteLine( x ) );
        transformBlock.LinkTo( actionBlock, new DataflowLinkOptions {PropagateCompletion = true});
        for( var i = 9; i > 0; i-- )
            transformBlock.Post( i * 1000 );
        transformBlock.Complete();
        actionBlock.Completion.Wait();
        Console.ReadLine();
    }
}

This outputs

edited Oct 10 '18 at 08:19

answered Oct 10 '18 at 08:02

Haukinger

10,420
2
15
28

I could not find EnsureOrdered via VisualStudio ObjectBrowser, i am using .NET 4.6.2 & DataFlow v4.5.24.0. Although I found it here https://github.com/dotnet/corefx/blob/d0dc5fc099946adc1035b34a8b1f6042eddb0c75/src/System.Threading.Tasks.Dataflow/src/Base/DataflowBlockOptions.cs – user1912383 Oct 10 '18 at 08:11
Which nuget are you using? `System.Threading.Tasks.Dataflow`? – Haukinger Oct 10 '18 at 08:14
`System.Threading.Tasks.Dataflow` Assembly from `Microsoft.Tpl.Dataflow` nuget. – user1912383 Oct 10 '18 at 08:17
There's a difference between `System.Threading.Tasks.Dataflow` nuget (current) and `Microsoft.Tpl.Dataflow` nuget (old). – Haukinger Oct 10 '18 at 08:21
I was using this https://www.nuget.org/packages/Microsoft.Tpl.Dataflow/ but i found a newer one https://www.nuget.org/packages/System.Threading.Tasks.Dataflow/. and now it works...thanks – user1912383 Oct 10 '18 at 08:25

TPL Dataflow execution output order

1 Answers1