0

I have built a web service in Visual Studio 2008, and deployed it on IIS 7 running on Windows Server 2008 R2.

It has been extensively tested, handles all errors gracefully and logs any uncaught errors to a file using log4net.

The system normally runs perfectly, but occasionally (2 or 3 times a day) a fault occurs and screws up the application which needs an iisreset to get it working again.

When the fault occurs I get some random errors that are caught by my catch-all error log, such as:

  • Value cannot be null.
  • Could not convert from type 'System.Boolean' to type 'System.DateTime'.
  • Sequence contains no elements

These errors are raised on every request until I manually do an iisreset, but there is no sight of them when the system is running normally.

The web service is a stateless request-response application. Nothing is stored in the session. It could be load related, but I doubt it as there are only around 30 or 40 requests per minute.

This is proving very tricky to track down. I can't work out what is putting IIS into this bad state, and why it needs an iisreset to get it working again. Nothing is reported in the event log.

Can anyone suggest how to enable more extensive logging that might catch this fault?

Also, does anyone know what might be causing the problem?

BG100
  • 169
  • 1
  • 14
  • 1
    Not what you want to hear, I'm sure, but it sounds like these are uncaught messages from some aspect of the application; IIS and .net wouldn't start kicking boolean-to-date conversion errors for no reason. – Shane Madden Mar 15 '11 at 16:00
  • @Shane Madden: I know, I've had the same thoughts, but I can't work out why a fault on one stateless request-response would affect all subsequent requests until I do an iisreset. – BG100 Mar 15 '11 at 16:03
  • 1
    @BG100 - because the app's internal state is getting mangled somehow. Incidentally, try recycling the App Pool hosting the application instead of IISReset -ting - this causes the process to terminate and a new one to start on the next request, without wiping out the whole server. This would confirm failure is isolated to that app. Understanding why the application is trashing itself will require debugging of the application. Instead of IISReset -ting it, take a memory dump of the process at the time, and work with your application developers and/or Microsoft to try to find the cause. – TristanK Mar 15 '11 at 22:06
  • I agree with Shane and TristanK. This is application related. Do you use application variables or session state? Either could cause global impact. Using your debug info, try to find out the exact code where the errors occur and work backwards from there. The 1st and 3rd error suggest missing data, which could mean that an application object you expect isn't available. The 2nd error sounds like a casting issue that may occur from incorrectly formatted data data. – Scott Forsyth Mar 16 '11 at 20:48
  • @TristanK & @Scott Forsyth: I'm not sure that it is application related. As I mentioned in my question, it is a stateless application that does not use session or application variables. When I get the errors, they don't seem to relate to anything in the code. From the my own debug logs, I can see the exact part of code where the error happens, yet I'm getting a boolean/datetime conversion error even though I'm not using boolean's and datetime's. – BG100 Mar 16 '11 at 21:27
  • @BG100 : I'm not blaming the app, just noting that the app's state is what's getting blatted. for this to happen in managed code usually suggests a bug somewhere in the code itself, or that the process hosting the code has been destabilized somehow (often as a result of native components behaving badly, or an exception path that leaves something not-cleaned-up properly). Same advice applies - you'll need to grab a memory dump when it's in the bad state, and try to debug why the structures aren't being populated properly. Right-click Create Dump in Taskman process view would do. – TristanK Mar 17 '11 at 03:59
  • @TristanK: Ok, that makes sense... so when I get a memory dump, how do I debug it? Can I load this into Visual Studio somehow and look at the problem? – BG100 Mar 17 '11 at 11:44
  • Short version: yes, but unless you've done it before and/or know what you're looking for, I'd suggest sending it to Microsoft. – TristanK Mar 17 '11 at 20:31

1 Answers1

0

I have fixed this problem myself. It was indeed an application bug.

To see the full explanation on how to resolve this if you get the same problem, please see my coresponding question and answer over on StackOverflow.

Community
  • 1
BG100
  • 169
  • 1
  • 14