Parallel For causes massive lag spikes after 2,3 minutes

Question

Edit:
I have noticed these lag spikes only occur while debugging in visual studio. If I run the .exe outside of Visual Stduio,the program doesnt use more than 3% of the CPU.Can anyone tell me why this is happening?

I have encountered a problem with Parallel processing.I'm using Parallel.For to check a large number of proxies (by making webrequests).This is my function:

private ConcurrentBag<string> TotalProxies = new ConcurrentBag<string>();
private void CheckProxies()
{
    ParallelOptions pOptions = new ParallelOptions();
    pOptions.MaxDegreeOfParallelism = 100;
    int max = TotalProxies.Count;
    Invoke(new Action(() => { lbl_Status.Text = "Checking"; }));
    Parallel.For(0, max, pOptions, (index, loopstate) =>
    {
        string Proxy = TotalProxies.ElementAt(index);
        if (WebEngine.IsProxyWorking(Proxy))
        {
            WorkingProxies.Add(Proxy);
            workingp++;
            Invoke(new Action(() =>
            {
                lstv_Working.Items.Add(Proxy);
                lbl_Working.Text = workingp.ToString();
            }));
        }
        checkedp++;
        Invoke(new Action(() => { lbl_Checked.Text = checkedp.ToString(); }));

        if (Stop)
            loopstate.Stop();
    });
    Invoke(new Action(() => {
        lbl_Status.Text = "Idle";
    }));
}

My problem is the following:
The program works fine for the first 0-2000 requests,where the cpu usage is around 3-5%.Then,after 2-3 minutes, I have encountered massive and frequent lag spikes,causing the CPU usage to jump up to 100%. I have no idea why this is happening,since it worked fine until now.I hope someone can help me understand what causes this.
Here you can see my problem:

Also,this used to work fine a few days ago.Now I have this problem without making any changes to the code.. — Catalin, Dec 28 '16 at 06:51
You can probably use a profiler to check which part of the code is holding the CPU. The fact that you haven't changed anything in your code since it was working alright leads me to think that one of the HttpRequest calls is probably behaving differently than it behaved a few days ago, most likely because something on the other side is behaving differently (website). — o_weisman, Dec 28 '16 at 07:23
@o_weisman How can I use a profiler ? I used the default diagnostic tool and what causes the most cpu usage,it says external code. What does that mean? — Catalin, Dec 28 '16 at 07:33
Also,there is no issue on the side ( website ) since I tried with different ones and I have the same issue — Catalin, Dec 28 '16 at 07:34
First a small side-note: `++` as in `checkedp++` is not atomic and can give wrong results in race conditions. It's safer to use `Interlocked.Increment`. That said, locking can cause hangups, although I don't know if they can be the cause of cpu spikes. But as long as you're invoking and not locking the variable itself, you could try running without the `lock`s. All in all, since the waiting is for IO, it could be better to use an `async...await` construction. That would also negate the need for a potentially high number of threads (`MaxDegreeOfParallelism`) — Me.Name, Dec 28 '16 at 07:40
@Me.Name Since I have never used async and await,can you guide me a little on how would I do it for my current need ? — Catalin, Dec 28 '16 at 07:46
I'll be away from keyboard for a while, but if I get back later in the day and you haven't gotten a better answer by then, I'll try to create a quick example if that's ok? — Me.Name, Dec 28 '16 at 08:23

Me.Name · Accepted Answer · 2016-12-28T14:26:44.750

As promised an example with async/await, although seeing your update I'm not sure if it will make a difference. But since it won't fit inside a comment, posted it here ;)

private ConcurrentBag<string> TotalProxies = new ConcurrentBag<string>();
private async Task CheckProxies()
{
    lbl_Status.Text = "Checking"; //NB, invoking is omitted assuming that CheckProxies is called from the UI thread itself
    var tasks = TotalProxies.Select(CheckProxy);
    await Task.WhenAll(tasks);
    lbl_Status.Text = "Idle";
}

private async Task<bool> CheckProxy(string p)
{   
    bool working = await Task.Run(() => WebEngine.IsProxyWorking(p)); //would be better if IsProxyWorking itself uses async methods and returns a task, so Task.Run isn't needed. Don't know if it's possible to alter that function?
    if(working)
    {
        WorkingProxies.Add(p);
        workingp++; //Interlocked.Increment is not necessary because after the await we're back in the main thread
        lstv_Working.Items.Add(p);  //are these items cleared on a new run? 
        lbl_Working.Text = workingp.ToString();
    }
    checkedp++;
    lbl_Checked.Text = checkedp.ToString(); 
    return working;
}

Note, since I couldn't the test the actual code, I'm not sure about the efficiency. Your current code might perform better. But if the IsProxyWorking method could use actual async webcalls (I believe that code was previously included in your post), I believe the processing could really improve.

If you made `CheckProxies` return `async Task` instead of `async void` I would upvote you. — Scott Chamberlain, Dec 28 '16 at 14:22
@ScottChamberlain That alone is reason for me ;) Changed the signature — Me.Name, Dec 28 '16 at 14:30
Thanks a lot for all your help. After doing some research,it seems the lag spikes occur in visual studio due to the large number of exceptions being thrown at once( not 100% sure though). — Catalin, Dec 28 '16 at 18:57

score 0 · Answer 2 · edited Jun 20 '20 at 09:12

I don't know if this is directly related to your issue, but setting the MaxDegreeOfParallelism to 100 is not good. You're basically telling your application to perform 100 tasks at the same time! According to MSDN

Generally, you do not need to modify this setting. However, you may choose to set it explicitly in advanced usage scenarios such as these:

When you know that a particular algorithm you're using won't scale beyond a certain number of cores. You can set the property to avoid wasting cycles on additional cores.

When you're running multiple algorithms concurrently and want to manually define how much of the system each algorithm can utilize. You can set a P:System.Threading.Tasks.ParallelOptions.MaxDegreeOfParallelism value for each.

When the thread pool's heuristics is unable to determine the right number of threads to use and could end up injecting too many threads. For example, in long-running loop body iterations, the thread pool might not be able to tell the difference between reasonable progress or livelock or deadlock, and might not be able to reclaim threads that were added to improve performance. In this case, you can set the property to ensure that you don't use more than a reasonable number of threads.

I would try to remove this value and see how your application behaves!

Parallel For causes massive lag spikes after 2,3 minutes

2 Answers2