How TPL parallelism works?

Question

I was experimenting with TPL blocks and came across a point where it puzzled me. TPL blocks shall run in parallel, depending on my settings (MaxDegreeOfParallelism).

What I have done?

I have created 5 blocks:

• Block 1 takes 3 seconds to complete

• Block 2 takes 1 second

• ...and rest completes in few milliseconds

All of them has MaxDegreeOfParallelism set to 32 (which is Environment.ProcessorCount X 4), EnsureOrdered set to false. All works good, every things runs in parallel. I am assuming this is the maximum throughput I shall expect. I am posting 50 messages in my block-chain asynchronously. A simple for loop posting integer value incrementally.

Now, I made one change, to Block 1. I added a setting in my ExecutionDataflowBlockOptions

TaskScheduler = new LimitedConcurrencyLevelTaskScheduler(2)

Now this setting is to limit my Block 1 to two threads only. It can get maximum of two threads only when running in parallel. Here is the code:

class Example1
{
    static void Main(string[] args)
    {
        Func<string, string> f = (string uri) =>
        {
            Thread.Sleep(3000);
            Console.WriteLine(DateTime.Now.ToString() + $">>[{AppDomain.GetCurrentThreadId()}] block1...({uri})");
            return uri.ToString(); ;
        };
        var block1 = new TransformBlock<string, string>(f, new ExecutionDataflowBlockOptions
        {
            EnsureOrdered = false,
            MaxDegreeOfParallelism = 32,
            TaskScheduler = new LimitedConcurrencyLevelTaskScheduler(2)
        });

        var block2 = new TransformBlock<string, string[]>(text =>
        {
            Thread.Sleep(1000);
            Console.WriteLine(DateTime.Now.ToString() + $">>[{AppDomain.GetCurrentThreadId()}] block2...({text})");
            var retVal = new string[1];
            retVal[0] = text;
            return retVal;
        }, new ExecutionDataflowBlockOptions
        {
            EnsureOrdered = false,
            MaxDegreeOfParallelism = 32,
        });

        var block3 = new TransformBlock<string[], string[]>(words =>
        {
            Console.WriteLine(DateTime.Now.ToString() + $">>[{AppDomain.GetCurrentThreadId()}] block3..({words[0]})");
            return words;
        }, new ExecutionDataflowBlockOptions
        {
            EnsureOrdered = false,
            MaxDegreeOfParallelism = 32
        });

        var block4 = new TransformManyBlock<string[], string>(words =>
        {
            Console.WriteLine(DateTime.Now.ToString() + $">>[{AppDomain.GetCurrentThreadId()}] block4...({words[0]})");
            var retVal = new ConcurrentQueue<string>();
            retVal.Enqueue(words[0]);
            return retVal;
        }, new ExecutionDataflowBlockOptions
        {
            EnsureOrdered = false,
            MaxDegreeOfParallelism = 32
        });

        var block5 = new ActionBlock<string>(reversedWord =>
        {
            Console.WriteLine(DateTime.Now.ToString() + $">>[{AppDomain.GetCurrentThreadId()}] block5...({reversedWord[0]})");
        }, new ExecutionDataflowBlockOptions
        {
            EnsureOrdered = false,
            MaxDegreeOfParallelism = 32
        });

        block1.LinkTo(block2);
        block2.LinkTo(block3);
        block3.LinkTo(block4);
        block4.LinkTo(block5);


        block1.Completion.ContinueWith(t =>
        {
            if (t.IsFaulted) ((IDataflowBlock)block2).Fault(t.Exception);
            else block2.Complete();
        });
        block2.Completion.ContinueWith(t =>
        {
            if (t.IsFaulted) ((IDataflowBlock)block3).Fault(t.Exception);
            else block3.Complete();
        });
        block3.Completion.ContinueWith(t =>
        {
            if (t.IsFaulted) ((IDataflowBlock)block4).Fault(t.Exception);
            else block4.Complete();
        });
        block4.Completion.ContinueWith(t =>
        {
            if (t.IsFaulted) ((IDataflowBlock)block5).Fault(t.Exception);
            else block5.Complete();
        });

        var d1 = DateTime.Now;
        for (int i = 0; i < 50; i++)
        {
            block1.SendAsync(i.ToString());
        }


        block1.Complete();
        block5.Completion.Wait();
        var d2 = DateTime.Now;
        Console.WriteLine($"Time taken {(d2.Ticks - d1.Ticks) / 10000000} seconds");
        Console.WriteLine("DONE...");
        Console.ReadLine();
    }

When I ran this code considering,

(t1, t2 represents threads)

(B1(1) represents block 1 executing message 1)

(B2(1) represents block 2 executing message 1)

(B1(2) represents block 1 executing message 2)

(T1, T2 represents unit of time)

        T1          T2
t1      B1(1)       B1(3)...

t2      B1(2)       B1(4)...

t3                  B2(1)

t4                  B2(2)

In time slot T2, my B2 block shall start running. But this is not the case:

9/11/2017 4:48:55 PM>>[1240] block1...(1)
9/11/2017 4:48:55 PM>>[9688] block1...(0)
9/11/2017 4:48:58 PM>>[9688] block1...(3)
9/11/2017 4:48:58 PM>>[1240] block1...(2)
9/11/2017 4:49:01 PM>>[1240] block1...(5)
9/11/2017 4:49:01 PM>>[9688] block1...(4)
9/11/2017 4:49:04 PM>>[1240] block1...(6)
9/11/2017 4:49:04 PM>>[9688] block1...(7)
9/11/2017 4:49:07 PM>>[9688] block1...(9)
9/11/2017 4:49:07 PM>>[1240] block1...(8)
9/11/2017 4:49:10 PM>>[1240] block1...(11)
9/11/2017 4:49:10 PM>>[9688] block1...(10)
9/11/2017 4:49:13 PM>>[9688] block1...(13)
9/11/2017 4:49:13 PM>>[1240] block1...(12)
9/11/2017 4:49:16 PM>>[1240] block1...(15)
9/11/2017 4:49:16 PM>>[9688] block1...(14)

Its running Block 1 for all the messages first and then starts Block 2.

Am I doing something wrong here?

or Is my expectation incorrect from TPL blocks (Pipes and Filters pattern)?