6

I'm managing a web site which is running fine for last couple of months over IIS 7.5 built with MVC 3.0 ASP.net. Every now and then we are facing a problem when our AJAX POST request (fired through jQuery) fails as the JSON being posted is getting truncated.

What we've found so far is that for all such request, the "Content-Length" header of the request has more data than we are actually getting in the request.

We've already set maxRequestLength to 51200 in our web.config and I believe the value of maxAllowedContentLength has a pretty large default value (we've not set that in our configuration). Also I have a failed request with "Content-Length" as low as 7301 (bytes) but we managed to get only 2179 bytes of it. So I'm not suspecting this to be hitting any limit.

Request Headers for a problematic request are as follows

  • Cache-Control: no-cache
  • Connection: keep-alive
  • Pragma: no-cache
  • Content-Length: 7301
  • Content-Type: application/json; charset=utf-8;
  • Accept: application/json, text/javascript, /; q=0.01
  • Accept-Encoding: gzip, deflate
  • Accept-Language: en-us,en;q=0.5
  • User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:15.0) Gecko/20100101 Firefox/15.0.1
  • X-Requested-With: XMLHttpRequest

Any ideas ??

Update : I've been able to further isolate the problem from our code. Have written an independent controller that accepts a JSON string and just deserializes it. In case of an error, it logs the error.

When I hit this controller in parallel with 150 concurrent threads in a loop of 50 requests then I get a few failures where the JSON that this controller receives is truncated. Now we are strongly focussed on optimizing IIS and reading more about various parameters that may be related (currently we are running with default parameters on IIS).

I strongly feel that 150 concurrent connections should not be a big deal and am sincerely hoping that tuning some parameters we should be able to get past this problem. Once we get past this problem, will share my findings.

*Update 2 (October 8) * : I have further narrowed down the problem. I turned on error log in IIS and discovered that my failed request gets the following error while reading data

BytesReceived = 0
ErrorCode = 2147943395
Error Description = "The I/O operation has been aborted because of either a thread exit or an application request.(0x800703e3)"

I am finding information about this error on iis forums but am yet to experiment with (mutliple) suggestions being given. The following link can be a good starting point for searching more on this

http://forums.iis.net/p/1149787/1917288.aspx

Guneet Sahai
  • 81
  • 1
  • 2
  • 5
  • Check this http://stackoverflow.com/questions/10966328/the-json-request-was-too-large-to-be-deserialized/10969382#10969382 – VJAI Sep 30 '12 at 16:08
  • Do you have any idea where the content is being truncated? Is it in the browser before it gets to the Ajax call? Is it as it's written written to the Ajax call? Is it in the HTTP transfer? Is it happening in some server-side script? – pieman72 Oct 01 '12 at 01:11
  • Thanks for responding guys. @Mark - I don't think this is related to size of JSON (I'm not getting max size error while deserialzing). Unfortunately I don't know where the data is getting truncated. I know that the controller is getting truncated data. I also know that most of the times when this problem occurs, the browser in use is IE 8. I'm still debugging and hopefully get to the bottom of this issue soon - will post my findings. – Guneet Sahai Oct 01 '12 at 11:20
  • **Update**: I've been able to reproduce the problem with JMeter but it only shows up with more than 100 concurrent threads hitting the server in a workflow. – Guneet Sahai Oct 01 '12 at 14:02
  • You can take a look at the [max concurrent connections](http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/36498ddb-695e-4aeb-87e5-c37df8fc48cc.mspx?mfr=true). That shouldn't cause truncation, but since you mentioned it, it's probably worth investigating. Small discussion on the topic [here](http://social.msdn.microsoft.com/Forums/en-US/architecturegeneral/thread/708fe47e-01e6-4c2b-9ea1-4bc483815d23). – pieman72 Oct 02 '12 at 05:02
  • Thanks @pieman72 for your link. Have updated new findings as an edit in the question – Guneet Sahai Oct 03 '12 at 14:38

1 Answers1

1

I finally figured out the cause and the fix for the solution. Unfortunately it is not applicable for all my environments but helps the production environment. Find details below

This turns out to be due to a bug in Windows 2008 R2, where it makes asp.net to believe that a client has disconnected even when it hasn't. A hotfix for the same is available and can be found at http://support.microsoft.com/kb/977453. The fix is already part of Windows 2008 R2 SP1 (found from here).

While the hotfix worked for me on one environment running Windows 2008 R2, it can not be applied to Windows 2008 R2 with SP1. Unfortunately the problem can still be reproduced on the environments running on SP1 and remains un-resolved. I have opened a new case on IIS forums for this issue at http://forums.iis.net/t/1192310.aspx and will track it there instead.

To read more about this issue - you can follow the thread at http://forums.iis.net/p/1149787/1917288.aspx

Guneet Sahai
  • 81
  • 1
  • 2
  • 5
  • 1
    We're having the same kind of error, but on Windows 2008 (not R2). Also, I'm confused, you said the "fix is already part of Windows 2008 R2 SP1" and then you state "unfortunately the problem can still be reproduced on the environments running on SP1." ? – Mason G. Zhwiti Feb 25 '13 at 16:42
  • did you get this fixed? I'm having a very similar problem... on Windows 2008 R2 SP1 the issue remains... on Windows 2008 R2, I apply the hotfix and everything is fine. – Tiago Coelho Aug 11 '15 at 14:49