Waiting on GET response

Question

I am having a problem where my code waits a long period of time on HttpWebResponse response = (HttpWebResponse)request.GetResponse(); What am I doing wrong? Sometimes I get bad proxies from my queue so I want to be able to timeout and skip.

        Boolean match = false;
        String clean = String.Empty;
        String msbuff;
        IWebProxy insideProxy;
        try
        {
            //grab proxies based on queue
            String proxyQueItem = Proxy_Que(_rankingConnectionString);
            insideProxy = new WebProxy(proxyQueItem);

            HttpWebRequest request = (HttpWebRequest) WebRequest.Create(ResultURL);
            request.Proxy = insideProxy;
            request.Timeout = 15*1000;

            using (HttpWebResponse response = (HttpWebResponse) request.GetResponse())
            {
                if (response.StatusCode == HttpStatusCode.OK)
                {
                    Stream streamResponse = response.GetResponseStream();
                    StreamReader streamRead = new StreamReader(streamResponse);
                    msbuff = streamRead.ReadToEnd();
                    //get rid of , -, (, )
                    msbuff = msbuff.Replace(" ", String.Empty).Replace(" ", String.Empty);
                    msbuff = msbuff.Replace("-", String.Empty);
                    msbuff = msbuff.Replace("(", String.Empty);
                    msbuff = msbuff.Replace(")", String.Empty);
                    //attempt to find phone number
                    match = msbuff.Contains(Listing.PhoneNumber);
                    streamRead.Close();
                }

            }
        }
        catch (Exception lk)
        { }

Any reason you're not using [WebClient](http://msdn.microsoft.com/en-us/library/system.net.webclient.aspx) (.NET 1.1 - 4.0) or [HttpClient](http://msdn.microsoft.com/en-us/library/system.net.http.httpclient.aspx) (.NET 4.5)? — dtb, May 02 '12 at 21:14
I also tried WebClient and it would wait on client.DownloadString — Charlie Morrison, May 02 '12 at 22:04
Are you asking about a specific error, and if so, what's the error? Or are you asking for suggestions on control of flow in order to skip to the next proxy in your queue? — Kevin Collins, May 02 '12 at 23:24
well basically my code is unusable due to waiting on get response ;s — Charlie Morrison, May 03 '12 at 00:18

score 2 · Answer 1 · answered Aug 21 '13 at 12:11

I ran into this same issue. The problem is with your timeout code. You have:

request.Timeout = 15*1000;

You need:

request.Timeout = 15*1000;
request.ReadWriteTimeout = 15*1000;

The .Timeout attribute is for the timeout on the send. However, after the request is sent and the reading has begun, there's an overall timeout for the request. It's set via the ReadWriteTimeout attribute.

The default value for the ReadWriteTimeout is 5 minutes, which is why you're seeing the behavior you are seeing.

score 1 · Answer 2 · answered May 02 '12 at 21:20

Since you're specifying a 15 second timeout, I'm assuming that you're waiting longer than that 15 seconds. The reason for the delay in your code could be that it's waiting for a connection to become available. The time spent waiting for a connection is unrelated to the timeout for your request.

I believe that by default, .NET allows only two simultaneous connections to an "endpoint" (IP address). This can be configured through your app/web config:

<system.net>
  <connectionManagement>
    <add address="www.example.com" maxconnection="10"/>
  </connectionManagement>
</system.net>

However, this might not solve the problem, because the server you're communicating with may only allow a limited number of connections per client. Your proxy may also be involved.

I am not sure if this is the problem. The problem I believe has to do with a bad proxy causing the getresponse to hang for 4-5mins. — Charlie Morrison, May 02 '12 at 22:10

score 1 · Answer 3 · answered Jun 26 '12 at 16:02

I ran into the problem before and a simple change to the timeout setup fixed it. This is VB.NET not C# but you should be able to figure it out. First, specify a second timeout method that can be threaded:

Private Sub TimeoutCallback(ByVal state As Object, ByVal timedOut As Boolean)
        If timedOut Then
            Dim request As Net.HttpWebRequest = state
            If Not (request Is Nothing) Then
                request.Abort()
            End If
        End If
    End Sub

Then when you build your request and read back you can setup something like this:

Try
            Dim myHttpWebRequest1 As Net.HttpWebRequest
            ' Create a new HttpWebrequest object to the desired URL.
            myHttpWebRequest1 = CType(WebRequest.Create(String.Format("http://{0}:{1}", inURL, inPORT)), Net.HttpWebRequest)

            ' Create an instance of the RequestState and assign the previous myHttpWebRequest1
            ' object to it's request field.  
            Dim myRequestState As New RequestState()
            myRequestState.request = myHttpWebRequest1
            ' Start the Asynchronous request.
            allDone.Reset()
            Dim result As IAsyncResult = CType(myHttpWebRequest1.BeginGetResponse(AddressOf RespCallback, myRequestState),  _
                                               IAsyncResult)
            ThreadPool.RegisterWaitForSingleObject(result.AsyncWaitHandle, _
                                                   New WaitOrTimerCallback(AddressOf TimeoutCallback), _
                                                   myHttpWebRequest1, DefaultTimeout, True)
            allDone.WaitOne()

Notice the TheadPool line (2nd from bottom), that will submit your timeout method on a separate thread so it can cancel the request even if your other request is hung up due to an invalid ip address or host.

score 0 · Accepted Answer · answered May 03 '12 at 00:58

If you have a queue of proxies and you just need to timeout and skip to the next one if it's bad, then you could just loop until you get a response, catching the error and getting a new proxy off the queue each time through. Something like this ...

private string requestByProxy(string url)
{
    string responseString, proxyAddress;

    while (responseString == null)
    {
        try
        {
            HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
            // set properties, headers, etc

            proxyAddress = getNextProxy();

            if (proxyAddress == null)
                throw new Exception("No more proxies");

            request.Proxy = new WebProxy(proxyAddress);
            request.Timeout = 15 * 1000;

            HttpWebResponse response = (HttpWebResponse)request.GetResponse();
            StreamReader sr = new StreamReader(response.GetResponseStream());
            responseString = sr.ReadToEnd();
        }
        catch (WebException wex)
        {
            continue;
        }
    }

    return responseString;
}

Waiting on GET response

4 Answers4