How to make a IEnumerable method parallel method

Question

Following to this post, I want parallelize this method :

    public IEnumerable<string> GetAllLogs(IEnumerable<IComputer> computers)
    {
        foreach (var cpt in computers)
        {
            foreach (var log in cpt.GetLogs())
            {
                yield return log;
            }
        }
    }

I want the method "yield returns" a log when one of the method GetLogs is finished. If I have 4 computers which returns :

Computer 01 : "a", "b", "c", "d", "e"
Computer 02 : "1", "2", "3", "4", "5"
Computer 03 : "alpha", "beta", "gamma", "delta", "epsilon"
Computer 04 : "I", "II", "III", "IV", "V"

With the "sequential method", the output is :

a
b
c
d
e
1
2
3
4
5
alpha
beta
gamma
delta
epsilon
I
II
III
IV
V

And the methods runs in 20 seconds. there is a Thread.Sleep(1000) in the GetLogs method.

I want the output looks like this :

III
a
4
gamma
b
c
IV
5
d
II
beta
e
1
2
delta
alpha
3
epsilon
I

and runs in few seconds.

I want to the methods returns an IEnumerable

score 20 · Accepted Answer · edited Mar 27 '18 at 12:18

20

This is what you need:

public IEnumerable<string> GetAllLogsParallel(IEnumerable<IComputer> computers)
{
    return computers
        .AsParallel()
        .SelectMany(cpt => cpt.GetLogs());
}

If you want to start processing of 4 computer-s at the same time you can adjust the degree of parallelism like this:

public IEnumerable<string> GetAllLogsParallel(IEnumerable<IComputer> computers)
{
    return computers
        .AsParallel()
        .WithDegreeOfParallelism(4)
        .SelectMany(cpt => cpt.GetLogs());
}

Following is a simplified explanation just for understanding. To learn more about that just visit PLINQ (Parallel LINQ) at MSDN.

Well, .AsParallel() - splits the computers enumerable into 4 parts and starts 4 threads at the same time. Each thread executes cpt.GetLogs() for each computer. The result is IEnumerable<IEnumerable<string>> - enumerable of enumerables. SelectMany() is used to flatten this list by concatenating inner enumerables and eliminating outer ones. Results are merged back automatically into the main thread in the order of their arrival.

edited Mar 27 '18 at 12:18

Yoh Deadfall

2,711
7
28
32

answered Nov 24 '11 at 14:22

George Mamaladze

7,593
2
36
52

Yeah, it works fine... All GetLogs() seems to finish in same time. (5 seconds env.) Yes I need explanations if possible :$ – Florian Nov 24 '11 at 14:28
That's right, but they wait for each other before you get any result. ( As a little side note ) – Felix K. Nov 24 '11 at 14:32
@FelixK. No they don't wait for each other. – Florian Nov 24 '11 at 14:35
@Florian Of course they do, not when executing the tasks, but the method returns all values at once and not with a behavior which is given by `yield`. – Felix K. Nov 24 '11 at 14:42
2

@FelixK. The return value of the method is an enumerable not the values returned by `GetLogs`. The method returns immediately. The values are queried and returned by enumerable as soon as you iterate it. – George Mamaladze Nov 24 '11 at 14:47
Good to know, i thought he parallizes the `GetLogs` and returns after everything is read. – Felix K. Nov 24 '11 at 14:51
it prints all at once. is this expected behaviour ? – Furkan Gözükara Apr 01 '13 at 12:37
printing at once is different behaviour from yield this example does not work. It is valid if you want the whole IEnumerable returned. Yield syntax implies returning one member at a time – John Nicholas Aug 07 '13 at 12:53

How to make a IEnumerable method parallel method

1 Answers1

Linked

Related