0

I am trying to make an HttpWebRequest to a remote sensor with an HTML server on it to pull back the data.

I think the problem is that the web server has some issue with it. I can get the web page to render in a web browser and the data is visible. That's the data I want to parse and store into a relational database.

When I look at the response headers in Fiddler, I can see that it is returning a 500 Bad Response. So, what I think is going on is, the headers say Bad Response but in includes the valid web page anyway.

I don't think i can fix the web server. I would like to just grab the data and parse it. Unfortunately the Response object is null.

The error thrown is WebExceptionStatus.ServerProtocolViolation and in the catch branch, the Response is null.

What can I do in this scenario?

UPDATE:

Another issue is that there is no HTML tag. Here is the raw returned response from Fiddler: 

HTTP/1.1 500 Fiddler - Bad Response
Date: Thu, 06 Aug 2015 18:13:27 GMT
Content-Type: text/html; charset=UTF-8
Connection: close
Cache-Control: no-cache, must-revalidate
Timestamp: 14:13:27.775

<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
<meta http-equiv="refresh" content="5"/>
<meta http-equiv="cache-control" content="no-cache">
<title>IQS WebServer</title>
<style type="text/css">
    body {font-family: Tahoma, Verdana, Arial, sans-serif; font-size: 0.8em;}
    table {/*font-size: 0.8em;*/ /*font-family: monospace;*/}
    .CaptionRow {font-weight:bold; background: #EFF3EB;}
    .OddRow {background: #EFF3EB;}
</style>
</head>

And there is a body from there.

I don't know why its throwing the 500 and why there is not an HTML tag but I don't really care about how poorly formed the response is - I really just want to get at the response and process the data contained within it regardless but the library isn't allowing it. The response is null and I am thus, stuck.

tnktnk
  • 512
  • 2
  • 7
  • 20
  • Oh and I have tried setting KeepAlive = false. I have also tried switching the ProtocolVersion to Version10 and Version11 both and nothing works. Those seem to be the usual tricks that work for others. – tnktnk Aug 06 '15 at 18:26
  • There's a Response property on the WebException. I can't remember precisely, but it might be that in error conditions that will be where the response goes. – Steve Howard Aug 06 '15 at 18:31
  • Right, so once you are in the catch branch, you have to cast e.Response like this: (HttpWebResponse)e.Response. My problem is that response is null. It's just not there. Except it really is there but the library has chosen to leave it null. – tnktnk Aug 06 '15 at 18:35

1 Answers1

0

https://msdn.microsoft.com/en-us/library/65ha8tzh(v=vs.110).aspx "By default, the .NET Framework strictly enforces RFC 2616 for URI parsing. Some server responses may include control characters in prohibited fields, which will cause the HttpWebRequest.GetResponse() method to throw a WebException. If useUnsafeHeaderParsing is set to true, HttpWebRequest.GetResponse() will not throw in this case; however, your application will be vulnerable to several forms of URI parsing attacks. The best solution is to change the server so that the response does not include control characters."

Another common related problem are servers that send only LF line endings, where HTTP requires CRLF. You have to look at the hex to tell the difference, it's not visible otherwise. useUnsafeHeaderParsing should also help here.

Tratcher
  • 5,929
  • 34
  • 44
  • I've set useUnsafeHeaderParsing = Yes in my app.config file and this does not make any different. I am unable to access the Response object because it is null. – tnktnk Aug 12 '15 at 02:33
  • I also set maximumErrorResponseLength = -1 and maximumResponseHeadersLength = -1 and that also had no impact. The response object is null. – tnktnk Aug 12 '15 at 02:47
  • Using Python this was all so easy. This was a loss for .NET today I am afraid. It should have been a simple task accomplished in minutes. – tnktnk Aug 15 '15 at 01:08