What makes this HTTPS WebRequest time out even though it works in the browser?

Question

Here's my request:

var request = (HttpWebRequest) WebRequest.Create("https://mtgox.com/");
request.CookieContainer = new CookieContainer();
request.AllowAutoRedirect = false;
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
request.Headers[HttpRequestHeader.AcceptEncoding] = "gzip, deflate";
request.Headers[HttpRequestHeader.AcceptLanguage] = "en-gb,en;q=0.5";
request.Headers[HttpRequestHeader.AcceptCharset] = "ISO-8859-1,utf-8;q=0.7,*;q=0.7";
request.Timeout = 5000;
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0) Gecko/20100101 Firefox/4.0";
request.Method = "GET";

request.GetResponse();

The headers were copied from Firefox using HttpFox. I used Fiddler2 to verify that at least for HTTP requests, the headers are completely identical between Firefox requests and my requests.

However, when performing a request to this specific website using HTTPS, the request simply times out. It works for other websites.

I must be performing it differently to Firefox, because it always works in Firefox. I can't debug it using Fiddler2, however, because whenever Fiddler2 forwards these requests they also time out, even when originated by Firefox.

Is it just a really buggy website? Which part of the above gives me away as not being Firefox?

score 43 · Accepted Answer · edited May 23 '17 at 11:46

43

Using Microsoft Network Monitor, I found that HttpWebRequest would get stuck at a stage where it's supposed to send back a client key exchange. It simply didn't. The server duly waited for it, but it never came.

What fixed it was forcing HttpWebRequest to use SSL3 instead of TLS (even though TLS is supposed to automatically turn into SSL3 if necessary):

ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3;

Why this is so I guess I'll never know - just one of those mysterious things that would take more time to figure out than anyone I know is willing to spend...

One thing that was different about the captures: the TLS variant had an "Alert" entry in the Server Hello response, which is absent from the SSL3 exchange and also from all the TLS exchanges that actually worked. Curiously, though, the same alert is present in a capture of Firefox performing the request successfully.

Finally, it appears that there was a temporary OCSP glitch just when I was first posting this question, which has since been resolved. This added to the mess, but isn't the core problem.

edited May 23 '17 at 11:46

Community

1
1

answered Apr 14 '11 at 23:43

enverpex

1,080
2
8
13

1

@enverpex - you magnificent bastard!! I have spend the last few hours kicking and screaming as to why for a specific site my HttpWebRequest is stalling, I tried all headers, this is the only code that worked, thank you!!!!! A++++++ – Dalbir Singh Apr 10 '13 at 19:55
God bless you, sir! This was happening when I was making two calls to the same server, on after the other. The second call would hang for unknown reasons until I made this change. – koopaking3 Oct 25 '13 at 17:28
This did not work for me. I had to set the SecurityProtocol to SecurityProtocolType.Tls12 | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls (see my anser below) – henon Sep 23 '16 at 20:38
1

How-to use ***Microsoft Network Monitor*** in this issue ? – Kiquenet Apr 05 '18 at 20:31

Prabu Arumugam · Answer 2 · 2013-09-15T05:36:15.620

12

The typical reason for timeout errors with HttpWebRequest/HttpWebReponse calls is "not closing the response object/stream", which keep the connections active. You have to simply close the Stream object or the HttpWebResponse object after reading the contents of the stream. Default connection-limit is 2.

HttpWebResponse webResponse = (HttpWebResponse)webRequest.GetResponse();
Stream stream = webResponse.GetResponseStream();
string responseString = ((TextReader)new StreamReader(stream)).ReadToEnd();
webResponse.Close();

Alternatively, if you implemented your logic in a web-service which will be concurrently invoked by many users, then you may want to consider increasing the connection-limit in HttpWebRequest object.

HttpWebRequest webRequest = (HttpWebRequest)HttpWebRequest.Create(url);    
webRequest.ServicePoint.ConnectionLimit = 20;

edited Sep 15 '13 at 05:36

answered Sep 15 '13 at 04:46

Prabu Arumugam

1,929
2
15
19

I came here for http:// (not https://) and not .close() my connections seems to have been the problem ie not worked in code. The pages always worked just loading them in the browser. – bendecko Jun 02 '15 at 09:51
1

That solved my problem. It's logical to close the stream after usage, similar to .dispose(). Good explanation, Thanks. Also the documentations says the same in Remarks section: [docs.microsoft HttpWebResponse](https://learn.microsoft.com/en-us/dotnet/api/system.net.httpwebresponse?view=net-5.0#remarks) – mihkov Nov 24 '20 at 08:59

score 9 · Answer 3 · edited May 23 '17 at 12:17

The .NET framework on Windows 7 implements a TLS extension: Server Name Indication (RFC4366). According to your post What does this TLS Alert mean the server is responding with "Unrecognized Name". Not sure why the connection is reported to time out, because it really doesn't. Your network traces should show that the client initiates a connection termination after this [FIN,ACK]. Downgrading to SSL3 avoids the invocation of the the SNI.

FYI: The same .NET framework on Windows XP does not use the TLS Server Name Indication Extension. Your program will work there....

In my case I traced the occurrence of this to a missing ServerName directive in Apache. Adding the ServerName to the SSL configuration solved it, because now the web server no longer is unaware of its name.

You can also turn off sending of SNI with Java 7: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7127374 — eckes, Jan 07 '12 at 12:46

score 4 · Answer 4 · answered Jul 26 '13 at 17:02

Recently found out about this same problem, and wanted to see if Microsoft might have fixed this in newer .net versions. This is the code I tested with (domain removed for privacy):

HttpWebRequest request =
    (HttpWebRequest)WebRequest.Create("https://www.xxSomeDomainNamexx.com/");
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream());
Console.WriteLine(reader.ReadToEnd());

Compiled this against .net versions 2.0, 3.0, and 3.5 and all seemed to exhibit the exact same behavior as everyone here has been discussing. Then I tried compiling against 4.0 and 4.5, and received proper responses in both places.

Based on this, it seems Microsoft has likely fixed this problem in newer .net versions. For posterity I also tested the SecurityProtocol change suggested by enverpex and it worked great on 2.0, 3.0, and 3.5.

score 4 · Answer 5 · answered Sep 23 '16 at 20:36

I had the same timeout issue with https requests and none of the answers helped me. Just now I found a solution that resolved the problem for me. Note that this works only with .net framework 4.5 or higher.

System.Net.ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls;
var response = WebRequest.Create("https://yoursecurewebservice").GetResponse();
var body = new StreamReader(response.GetResponseStream()).ReadToEnd();
Console.WriteLine(body);

score 2 · Answer 6 · answered Apr 13 '11 at 18:48

2

Most likely the HTTPS client can't validate the certificate chain presented by this server, for example due to missing root certificate or inaccessible OCSP responder. I.e. there can be something different configured in the browser and in HTTPS client you use.

As one of the options you can take a trial version of our HTTPBlackbox components and try to connect using TElHTTPSClient. It will give you detailed error information (in case of error) so you will be able to determine, what's wrong with HttpWebRequest.

answered Apr 13 '11 at 18:48

Eugene Mayevski 'Callback

45,135
8
71
121

Hm, with Fiddler2 in the middle, connecting to google over HTTPS causes an exception about invalid certificates - not just hangs/timeouts... – enverpex Apr 13 '11 at 18:50
@enverpex I would bet on OCSP or CRL checking - they require additional HTTP(S) connections which can time out. But this is the thing you need to debug yourself - we can only guess here. – Eugene Mayevski 'Callback Apr 13 '11 at 19:03
Same thing happens using your HTTPBlackbox\HTTPGet sample for HTTPS/mtgox.com/443: "Connection failed". You said there should be detailed error information available, but I couldn't find much else inside the exception object. Could you hint at where to look? – enverpex Apr 14 '11 at 01:36
@enverpex this means that there's no connection to the site. Most likely it's some firewall that lets the browser pass but blocks your application. Also, if you handle OnError event, there can be extended information available. – Eugene Mayevski 'Callback Apr 14 '11 at 06:24
Using OnError, it turns out that this is a certificate validation problem. The exception thrown by TElHTTPSClient could really mention the error code too. I guess that, like you say, StartCom certificates can't be validated for some reason by .NET or your code, but can be by Firefox and Chrome. Even "https://www.startcom.org" itself causes "Connection failed" in HTTPBlackbox. – enverpex Apr 14 '11 at 10:49
@enverpex OnError reports SSL errors, while exception happens on the other level. – Eugene Mayevski 'Callback Apr 14 '11 at 10:51
Curious; HTTPBlackbox reports an "OCSP not verified" error for these sites, but even if I force Firefox to always use OCSP and to treat errors as failures, both sites still work in Firefox. – enverpex Apr 14 '11 at 11:18

What makes this HTTPS WebRequest time out even though it works in the browser?

6 Answers6

Linked