Parallel.For WebRequests - The first requests is just doing nothing

Question

I've got a list of Accounts. I want to login with all accounts on a website. I want to use Parallel.ForEach to process all accounts. This is what my code looks like:

Parallel.ForEach(Accounts,
    acc =>
    {
        acc.WebProxy = null;
        acc.Channel = "pelvicpaladin__";
        Debug.WriteLine("Connecting to {0}.", new object[] { acc.Username });
        acc.ConnectToIrc().Wait();
    });

Everything works fine except one single problem: The first Account in the Accounts-list does not work. Internal I have to use more than one request (it is a bit more than just logging in). The first request just does nothing. If I break the debugger, there is no available source.

I've got about 12 accounts. I've tried to remove the first account from the list. But the problem is still the same (now the new first (old second) account fails).

And now the very strange point: If I don't use Parallel.For, everything works fine.

foreach (var acc in Accounts)
{
    acc.WebProxy = null;
    Debug.WriteLine("Connecting to {0}.", new object[] { acc.Username });
    await acc.ConnectToIrc();
}

Again: Everything works except the first account from the list. It is always the first (it does not depend on how much accounts the list contains or which account is the first account).

Does anyone has any idea?

EDIT: This is how I create WebRequests:

private async Task<string> GetResponseContent(HttpWebRequest request)
{
    if (request == null)
        throw new ArgumentNullException("request");

    using (var response = await request.GetResponseAsync())
    {
        return await GetResponseContent((HttpWebResponse)response);
    }
}

private async Task<string> GetResponseContent(HttpWebResponse response)
{
    if (response == null)
        throw new ArgumentNullException("response");

    using (var responseStream = response.GetResponseStream())
    {
        return await new StreamReader(responseStream).ReadToEndAsync();
    }
}

private HttpWebRequest GetRequest(string url)
{
    if (String.IsNullOrWhiteSpace(url))
        throw new ArgumentNullException("url");

    try
    {
        HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
        request.CookieContainer = _cookieContainer;
        request.Referer = url;
        request.ContentType = "application/x-www-form-urlencoded; charset=UTF-8";
        request.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.4) Gecko/20091016 Firefox/3.5.4 GTB6 (.NET CLR 3.5.30729)";
        if (_webProxy != null)
            request.Proxy = WebProxy.WebProxy;

        request.KeepAlive = true;
        request.Timeout = 30000;

        return request;
    }
    catch (Exception ex)
    {
        ErrorLogger.Log(String.Format("Could not create Request on {0}.", url), ex);
        return null;
    }
}

I guess the issue is in the `ConnectToIrc` implementation. Can you share that code? — Wagner DosAnjos, Feb 11 '14 at 14:04
Well it is quite a lot of code. Many request but I can show you how I create the requests (btw. I've already checked whether all responses are getting disposed). — Florian, Feb 11 '14 at 14:30
That .Wait() is Task.Wait(), right? Doesn't seem required, remove it. Most likely the 1st action is executed on the calling thread , maybe it executes if you wait long enough? The first can be the last. — H H, Feb 11 '14 at 14:47

Servy · Accepted Answer · 2014-02-11T20:35:15.757

You're running into the typical await deadlock situation. The problem is that your calls to ConnectToIrc are using await and capturing the synchronization context. They're trying to marshall the continuations to the main thread. The problem is that your main thread is busy blocking on the call to Parallel.ForEach. It's not allowing any of those continuations to run. The main thread is waiting on the continuations to continue, the continuations are waiting on the main thread to be free to run. Deadlock.

This is (one reason) why you shouldn't be synchronously waiting on asynchronous operations.

Instead just start up all of the asynchronous operations and use WhenAll to wait for them all to finish. There's no need to create new threads, or use the thread pool, etc.

var tasks = new List<Task>();
foreach (var acc in Accounts)
{
    acc.WebProxy = null;
    Debug.WriteLine("Connecting to {0}.", new object[] { acc.Username });
    tasks.Add(acc.ConnectToIrc());
}
await Task.WhenAll(tasks);

This, unlike your second example, will perform all of the async operations in parallel, while still waiting asynchronously.

DrinkBird · Answer 2 · 2014-02-11T18:36:32.417

0

Updated again:

    var tasks = Accounts.Select(MyTask).ToList();
    await Task.WhenAll(tasks);

then you can write a named method:

    private Task MyTask(Account acc)
    {
        acc.WebProxy = null;
        Debug.WriteLine("Connecting to {0}.", new object[] { acc.Username });
        return acc.ConnectToIrc();
    }

thanks for the tip

edited Feb 11 '14 at 18:36

answered Feb 11 '14 at 14:38

DrinkBird

834
8
17

Parallel.ForEach often gives me a hard time too – DrinkBird Feb 11 '14 at 14:44
Well I don't know why. But was right. It works @HenkHolterman Still would like to know why. – Florian Feb 11 '14 at 14:52
1

Since you're not waiting on `ConnectToIrc` this will "finish" when the tasks all finish being started, not when they finish executing. – Servy Feb 11 '14 at 14:54
True, I just want to illustrate that you can have the whole async logic outside of your function - or within a lambda – DrinkBird Feb 11 '14 at 18:38

Parallel.For WebRequests - The first requests is just doing nothing

2 Answers2