How does PLINQ Order input

Question

If I have the following order in an IEnumerable (list) :

1 - 2 - 3 - 4 - 5

and If I run a PLINQ query on this say :

list.AsParallel().AsOrdered().WithDegreeOfParallelism(10).Select(
                s => SomeDelegateFunction(s)).AsSequential().ToList();

For the above query I see in my logging (inside delegate function) that it uses multiple threads, but does not maintain the sequence of processing of the list. However for the below query i does maintain the sequence but uses a single thread to do the entire operation:

list.AsParallel().AsOrdered().WithDegreeOfParallelism(10).AsSequential().Select(
               s => SomeDelegateFunction(s)).ToList();

The difference between the two queries is the "AsSequential()" in the second query, the question I had is that when I use AsSequential() :

1 - Why does it not use multi threads? It could have broken down the work as :

1 - 2 (Give it to thread 1)
3 - 4 - 5 (Give it to thread 2)

Instead it does do 1 - 2 - 3 - 4 - 5 (In that order) but does it on a single thread - why??

Basically I need to process my list in the ORDER it came into as input but on multiple threads.

Any ideas ?

Servy · Accepted Answer · 2012-11-30T19:47:13.463

4

Basically I need to process my list in the ORDER it came into as input but on multiple threads.

Those are mutually exclusive requirements. You can do one or the other, but never both.

If you don't care what order the items were processed in, you just want to make sure that your end result is that the objects are in the order that they first came in as you can do something like this:

var list = new List<int> { 1, 2, 3, 4, 5 };

Parallel.For(0, list.Count, i =>
{
    list[i] = Process(i);
});

If you prefer PLINQ over Parallel.For you could do something like this:

Enumerable.Range(0, list.Count)
    .AsParallel()
    .WithDegreeOfParallelism(10)
    .ForAll(i =>
    {
        list[i] = Process(i);
    });

edited Nov 30 '12 at 19:47

answered Nov 30 '12 at 19:29

Servy

202,030
26
332
449

Ah...Theoretically the division of work could have been done sequentially and then worked upon in parallel - correct? – Murtaza Mandvi Nov 30 '12 at 19:36
@MurtazaMandvi In the statement that I quoted you specifically say that they need to be processed in the order that they came in as. That makes the problem not solvable in parallel. If you want it to process them in any order, but ensure that when they are all done the final results are returned in the same order as when they were first received, then that's possible, but it's not what you asked. – Servy Nov 30 '12 at 19:39
Your code will work, but I think using `AsOrdered()` is a much cleaner alternative: you don't have to deal with indexes and the result will be the same. – svick Nov 30 '12 at 21:40
@svick I don't use PLINQ much, so I just did what I knew would work. If there is a more appropriate method based on PLINQ design by all means post it. – Servy Nov 30 '12 at 21:42
1

@Servy The more appropriate method is the first snippet in the question. But the OP doesn't like it because it “does not maintain the sequence of processing of the list”. Which, as you noted, doesn't make sense with parallel execution. – svick Nov 30 '12 at 21:44

score 1 · Answer 2 · answered Nov 30 '12 at 20:42

Answer to your first question

Why does it not use multi threads?

is that what AsSequential() does is exactly opposite to what AsParallel() does. AsParallel() makes a ParallelQuery<T> out on an ordinal enumerable IEnumerable<T> to allow parallel execution in multiple threads. AsSequential() is used to merge results of parallel execution into one enumerable back into the caller thread.

Answer to your second question

It could have broken down the work as :

1 - 2 (Give it to thread 1)
3 - 4 - 5 (Give it to thread 2)

Yes it could, but in case of parallel execution of this sequences 3 will probably process earlier then 1 and it will not be a sequential execution.

Nevertheless you can explicitly control how PLINQ breaks down the sequence into multiple partitions. So you can control yourself how your data gets distributed between multiple threads see Custom Partitioners for PLINQ

score 0 · Answer 3 · answered Nov 30 '12 at 19:46

A multithreaded producer-consumer pattern would allow you to ensure that the items are processed in the same order, while allowing items to be processed concurrently. If you need to ensure that each item's processing completes before the next item's processing starts, however, you're out of luck, as Servy says.

In the producer-consumer pattern, to maintain order, you can use a thread-safe queue. One or more producer threads enqueues items to be processed; in this case, you would have one producer passing the items from the list, in order, to the queue. Multiple consumer threads can then dequeue the items, which will be in order.

See http://en.wikipedia.org/wiki/Producer-consumer_problem for more information.

How does PLINQ Order input

3 Answers3