WebRequest timeouts when processing in parallel

Question

I'm just trying to understand how does processing in parallel work in c#. This is my code example:

Parallel.For(0, 1000, index =>
{
    try
    {
        var url = "some exemplary url";
        HttpStatusCode statusCode;
        HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
        StringBuilder data = new StringBuilder();

        using (HttpWebResponse myWebResponse = (HttpWebResponse)request.GetResponse())
        {
            statusCode = myWebResponse.StatusCode;
            if (myWebResponse.GetResponseStream() != null)
            {
                StreamReader streamReader = new StreamReader(myWebResponse.GetResponseStream());

                using (streamReader)
                {
                    string line = string.Empty;
                    while ((line = streamReader.ReadLine()) != null)
                    {
                        data.Append(line);
                    }
                }
            }
            MailMessage msg = new MailMessage();

            msg.Subject = "subject";
            msg.From = new MailAddress("no-reply@domain.co.uk");
            msg.To.Add("name.name@domain.com");
            msg.Body = data.ToString();
            msg.IsBodyHtml = true;
            var newSmtp = new SmtpClient();
            newSmtp.Send(msg);

            Interlocked.Increment(ref count);
        }
    }
    catch (Exception e)
    {
    }
});

Technicalities of the code are not that important here since it works fine. However, a request to "some exemplary URL" is fairly time-consuming and after processing +-30 requests I'm getting "Operation has timed out" exception thrown. When I change the Parallel for loop and add MaxDegreeOfParallelism option like so

Parallel.For(0, 1000, new ParallelOptions { MaxDegreeOfParallelism = 4 }, index =>
{

I no longer get the exception thrown. So, I'm trying to figure out why adding this option fixed the workflow? Looks to me that it's something to do with thread management? My understanding of it is that when I do not specify the number of threads, new ones are created before waiting for the previous one to finish? Am I right, or is it something wrong with my code?

Quentin Roger · Accepted Answer · 2016-05-26T11:53:32.037

2

There is a limit in the number of simultaneous outgoing HTTP connections. If you want to increase the number of threads you should increase the DefaultConnectionLimit before creating your HttpWebRequest.

System.Net.ServicePointManager.DefaultConnectionLimit

You can play with <connectionManagement> configuration tag https://msdn.microsoft.com/en-us/library/fb6y0fyc.aspx

<configuration>
  <system.net>
    <connectionManagement>     
      <add address = "*" maxconnection = "2" />
    </connectionManagement>
  </system.net>
</configuration>

edited May 26 '16 at 11:53

answered May 26 '16 at 11:46

Quentin Roger

6,410
2
23
36

thanks for your input, however, I still cannot get why setting MaxDegreeOfParallelism option fixed it? Does it mean that when I set its value to 4, this value is within the DefaultConnectionLimit, therefore it doesn't throw any exceptions? – Bartosz May 26 '16 at 12:12
Exactly when you set MaxDegreeOfParallelism to 4 you limit the number of simultaneous parallel `HttpWebRequest`. Anyway the server you try to access can limit the number of concurrent connections for an host that's why the connectionManagement exists. – Quentin Roger May 26 '16 at 12:24
I've checked my DefaultConnectionLimit and it's set to 2. So how does it cope with 4 simultaneous parallel web requests set in MaxDegreeOfParallelism option? – Bartosz May 26 '16 at 12:32
Its a funnel effect problem, when you increase the number of threads without increasing the number of possible connections overflowed after few seconds. – Quentin Roger May 26 '16 at 12:40

WebRequest timeouts when processing in parallel

1 Answers1