1

I'm writing a program that reads data from a series of web pages; this works for the first page, however it then hangs while waiting for a response from the next page until it times out. Whether I start by reading page 1, 2, 140, etc. it will always successfully read the first page but none after that.

I think this might be related to the "cookieHeader" variable, which is needed to access the pages as the website requires a user to login first. However when I inspect this element, its expiration time is set as the following day, so I don't see how it could be expired yet.

I am new to this so I'm hoping someone who has encountered this problem before or who has a better understanding of cookies could help me. I would appreciate any input! Below is a snippet of code, where the timeout error is being caught by the try-catch wrapper.

// loop through each page
for (int i = 1; i <= totalPages; i++)
{
    string thisUrl = chatUrl + i; //add page number to url
    WebRequest getReq = WebRequest.Create(thisUrl);
    getReq.Headers.Add("Cookie", cookieHeader);
    try
    {
        WebResponse getResp = getReq.GetResponse();
        Console.WriteLine("Page " + i + " read successfully");
    }
    catch (Exception e)
    {
        Console.WriteLine("Page " + i + " failed");
    }
}
user1959309
  • 101
  • 1
  • 11
  • It's just that sometimes requests fail. You can have a helper method to make a request and try-catch 3 times before ultimately rethrowing the exception, indicating that attempting again won't do you any good. – SimpleVar Feb 01 '15 at 01:05
  • It seems to be constant though; after the first page is read, all other pages fail and it takes some time before it fails so retrying can be very time consuming (it's currently taking approx. 7 minutes for 6 pages where 5 have failed, and I have 100+ pages to read) – user1959309 Feb 01 '15 at 01:16
  • 1
    Fiddler (http://www.telerik.com/download/fiddler) will help you sort out what is being sent and returned (or not). There's probably something wrong with the second GET and fiddler will help you see what you are missing. – Keith Payne Feb 01 '15 at 01:23

1 Answers1

2

I think it has got something to do with not closing the response properly. WebResponse implements IDisposable if I remember it right. So try disposing it using using statement. Also, there are some restrictions on some client versions of .Net where, by default, you can only have 2 connections open to the same server simultaneously. This restriction might kick in if you do not dispose the connections (WebResponse) properly (potentially because of keep alive settings).

user92454
  • 1,201
  • 1
  • 10
  • 13