Application Pool crashing after every automatic recycle since upgrade to ASP.NET 4.0

Question

After upgrading an ASP.NET solution to 4.0 we have noticed that every time the application pool recycles automatically (every 1740 minutes), exactly 32 seconds later we get an error in the event logs with event ID 5009.

I installed the debugger on the server and ran the analysis tool on the crash dump and below you can see what I got. Can you help me understand what could be the issue? This is happening on 4 different machines (which all happen to be virtual machines). I noticed that there is a "loop" going on in the full call stack (which I spare you here for space reasons), with the following group of calls being repeated over and over:

clr!Thread::DoAppropriateWaitWorkerAlertableHelper+4d 0000000000000000 000000001bb30ba0 0000000020a0cf60 0000000000000000
clr!Thread::DoAppropriateWaitWorker+146 000000001bb30ba0 0000000000000001 0000000001c05970 0000000000000000
clr!Thread::DoAppropriateWait+73 0000000020a0c0a0 0000000000000001 000007fee4736ca9 000007fee4736c2c
clr!CLREvent::WaitEx+c1 0000000000000000 0000000000000000 0000000000000000 000007fee7e7f960
clr!RegMeta::SaveToStream+134 0000000020a0b740 000000001bb30ba0 00000000dc951994 0000000076df0e4a
clr!WaitForEndOfShutdown+9 0000000000010000 000007fee7804b65 0000000000000000 0000000000000000
clr!Thread::RareDisablePreemptiveGC+490 000007fee4a8cca0 0000000020a0b9f0 00000000dc951994 0000000000001200
clr!EEHashTableBase,0>::FindItem+5c 000007fee4a8cca0 0000000020a0b9f0 0000000000000001 000007feefbc229b
clr!IsCerRootMethod+b1 0000000000000001 0000000000000000 0000000000000000 00000000ffffff98
clr!TAStackCrawlCallBack+1e9 0000000020a0bd90 0000000020a0b9f0 0000000020a0bd90 0000000020a0bd90
clr!Thread::MakeStackwalkerCallback+2f 0000000020a0c2a0 0000000000000002 0000000000000002 0000000020a0c060
clr!Thread::StackWalkFramesEx+8d 000000001bb30ba0 000000001bb30ba0 000000001bb30ba0 0000000000020019
clr!Thread::ReadyForAsyncException+197 0000000076f471ba 000000001bb30ba0 0000000000000000 0000000000000000
clr!Thread::HandleThreadAbort+6c 0000000076def5e4 000007fee4da009a 000000001bb30ba0 0000000000000000
clr!Thread::HandleThreadInterrupt+7e 00000000000007c0 00000000000007c0 000007fee79ae71c 000007fee4d9f9dc

Analysis Summary
Type Description Recommendation Error In w3wp_QAWEB4_PID_4668_Date__05_02_2012__Time_04_16_30PM_242_Second_Chance_Exception_C0000005.dmp the assembly instruction at clr!CLREvent::WaitEx+f in C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll from Microsoft Corporation has caused an access violation exception (0xC0000005) when trying to write to memory location 0x20990ff0 on thread 20 Please follow up with the vendor Microsoft Corporation for C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll

Information DebugDiag determined that this dump file (w3wp_QAWEB4_PID_4668_Date__05_02_2012__Time_04_16_30PM_242_Second_Chance_Exception_C0000005.dmp) is a crash dump and did not perform any hang analysis. If you wish to enable combined crash and hang analysis for crash dumps, edit the Globals.asp script (located in the DebugDiag\Scripts\inc folder) and set the g_DoCombinedAnalysis constant to True.

Analysis Details

Report for w3wp_QAWEB4_PID_4668_Date__05_02_2012__Time_04_16_30PM_242_Second_Chance_Exception_C0000005.dmp Type of Analysis Performed Crash Analysis Machine Name XXXX01-QA-WEB01 Operating System Windows Server 2008 Service Pack 2 Number Of Processors
Process ID 4668 Process Image c:\Windows\System32\inetsrv\w3wp.exe System Up-Time 4 day(s) 14:59:36 Process Up-Time 00:06:32

Thread 20 - System ID 4364 Entry point clr!Thread::intermediateThreadProc Create time 5/2/2012 4:10:16 PM Time spent in user mode 0 Days 0:0:1.421 Time spent in kernel mode 0 Days 0:0:0.453

.NET Call Stack

Function System.Threading.Thread.SleepInternal(Int32) System.Threading.Thread.Sleep(System.TimeSpan) Sitecore.Services.Heartbeat.WorkLoop()* ERROR: Module load completed but symbols could not be loaded for Sitecore.Kernel.dll System.Threading.ExecutionContext.runTryCode(System.Object) System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode, CleanupCode, System.Object) System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) System.Threading.ThreadHelper.ThreadStart()

Full Call Stack removed for space reasons

Exception Information CLR!CLREVENT::WAITEX+FIn w3wp_QAWEB4_PID_4668_Date__05_02_2012__Time_04_16_30PM_242_Second_Chance_Exception_C0000005.dmp the assembly instruction at clr!CLREvent::WaitEx+f in C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll from Microsoft Corporation has caused an access violation exception (0xC0000005) when trying to write to memory location 0x20990ff0 on thread 20

Module Information Image Name: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll Symbol Type: PDB Base address: 0x0000000300905a4d Time Stamp: Sat Jul 09 05:44:20 2011 Checksum: 0x0000000000000000 Comments: Flavor=Retail COM DLL: False Company Name: Microsoft Corporation ISAPIExtension: False File Description: Microsoft .NET Runtime Common Language Runtime - WorkStation ISAPIFilter: False File Version: 4.0.30319.239 (RTMGDR.030319-2300) Managed DLL: False Internal Name: clr.dll VB DLL: False Legal Copyright: © Microsoft Corporation. All rights reserved. Loaded Image Name: clr.dll Legal Trademarks:
Mapped Image Name: Original filename: clr.dll Module name: clr Private Build: DDBLD205 Single Threaded: False Product Name: Microsoft® .NET Framework Module Size: 9.39 MBytes Product Version: 4.0.30319.239 Symbol File Name: c:\symcache\clr.pdb\887DDF30473443C9905C3217115D7DE72\clr.pdb Special Build: &

Script Summary
Script Name Status Error Code Error Source Error Description Source Line CrashHangAnalysis.asp Completed

Is your process beeing profiled? I have seen call stack with RegMeta in their call stacks when a profiler was attached. — Alois Kraus, May 02 '12 at 22:37
Not sure what you mean by "being profiled". How do I check that? — Francesco Gallarotti, May 02 '12 at 22:40
Do you mean having perfmon running on the same machine, by any chance? — Francesco Gallarotti, May 02 '12 at 22:42
No I mean a memory or Performance Profiler like YourKit, Ants, or the VS Profiler, ... — Alois Kraus, May 03 '12 at 05:35
Not to my knowledge. I do have perfmon running continuously on the production servers, but the problem occurs also on the QA server which doesn't have perfmon running — Francesco Gallarotti, May 03 '12 at 11:05
It looks like Sitecore starts a new thread (for Heartbeat.Workloop) which never exits when w3wp wants it to. — sisve, May 10 '12 at 17:54
Franz, have you worked this around? Now I have pretty similar stack in WinDbg and looking for some ideas why it could happen. — KateButenko, Aug 31 '12 at 14:37

score 0 · Answer 1 · answered May 03 '12 at 13:48

0

What is error code listed in event id 5009 description? http://www.eventid.net/display-eventid-5009-source-Microsoft-Windows-WAS-eventno-8833-phase-1.htm indicates that this might be caused by an empty folder from the previous .Net version.

answered May 03 '12 at 13:48

AdiGri

41
3

Application Pool crashing after every automatic recycle since upgrade to ASP.NET 4.0

1 Answers1