I have 50+ asp.net 4.5 websites on 1 server(2012 R2), in 5 different App Pools. Occasionally I would met NullReferenceException when pool recycled and restarting with this stacktrace:
System.Web.HttpApplication.GetFactory(String type) +16
System.Web.MaterializeHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +374
System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +287
And this exception is gone if I modify web.config (adding/removing newlines in file end) for the website and send http request to it again.
I looked the source code for HttpApplication.GetFactory(string type):
private Hashtable _handlerFactories = new Hashtable();
....
private IHttpHandlerFactory GetFactory(string type) {
HandlerFactoryCache entry = (HandlerFactoryCache)_handlerFactories[type];
if (entry == null) {
entry = new HandlerFactoryCache(type);
_handlerFactories[type] = entry;
}
return entry.Factory;
}
and the source code for HttpApplication.MaterializeHandlerExecutionStep.IExecutionStep.Execute():
...
if (!String.IsNullOrEmpty(configType)) {
IHttpHandlerFactory factory = _application.GetFactory(configType);
string pathTranslated = request.PhysicalPathInternal;
...
I can't find anywhere a NullReferenceException could be thrown. So I use windbg with sos to debug, here is some useful information I get:
0:061> !eeversion
4.0.30319.34011 free
Server mode with 8 gc heaps
SOS Version: 4.0.30319.34011 retail build
0:061> !clrstack -a
OS Thread Id: 0x208c (61)
Child SP IP Call Site
0000006c11cdd590 00007ff97bb6d841 System.Web.HttpApplication.GetFactory(System.String)
PARAMETERS:
this = <no data>
type = <no data>
LOCALS:
<no data>
0000006c11cdd5d0 00007ff97bb93067 System.Web.HttpApplication+MaterializeHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
PARAMETERS:
this (0x0000006c11cdd690) = 0x0000006b4448d188
LOCALS:
0x0000006c11cdd630 = 0x00000067c515b380
<no data>
<no data>
<no data>
<no data>
<no data>
<no data>
<no data>
0x0000006c11cdd618 = 0x0000000000000000
0x0000006c11cdd610 = 0x0000000000000000
0x0000006c11cdd608 = 0x0000000000000000
<no data>
0000006c11cdd690 00007ff97bb6bf60 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)
PARAMETERS:
this (0x0000006c11cdd730) = 0x0000006b44489978
step = <no data>
completedSynchronously (0x0000006c11cdd740) = 0x0000006c11cdd822
LOCALS:
0x0000006c11cdd708 = 0x0000000000000000
0x0000006c11cdd700 = 0x0000000000000000
0x0000006c11cdd6f8 = 0x0000000000000000
0x0000006c11cdd6f0 = 0x0000000000000000
<no data>
......
0:061> r edx
edx=c5186090
0:061> !dumpobj edx
Name: System.String
MethodTable: 00007ff97e83fc48
EEClass: 00007ff97e1d1aa8
Size: 110(0x6e) bytes
File: C:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
String: System.Web.Handlers.TransferRequestHandler
0:061> !dumpobj 0000006b44489978
Name: ASP.global_asax
MethodTable: 00007ff92b6c2218
EEClass: 00007ff92b65ea50
Size: 224(0xe0) bytes
File: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\347f4708\a88050db\App_global.asax.v9t8txcb.dll
Fields:
MT Field Offset Type VT Attr Value Name
00007ff97bcf6348 4000fa5 8 ...pApplicationState 0 instance 0000006ac4269f28 _state
00007ff97bcf4a98 4000fa6 10 ...m.Web.HttpContext 0 instance 0000000000000000 _initContext
00007ff97bcf6fa8 4000fa7 18 ...b.HttpAsyncResult 0 instance 0000000000000000 _ar
00007ff97bcf6f38 4000fa9 20 ...pModuleCollection 0 instance 0000006b44489a58 _moduleCollection
00007ff97d67b558 4000fc5 28 ....EventHandlerList 0 instance 0000006b4448a578 _events
00007ff97bcfea28 4000fc6 30 ...ventHandlersTable 0 instance 0000006b4448ab20 _asyncEvents
00007ff97bd03070 4000fc7 38 ...ation+StepManager 0 instance 0000006b4448d160 _stepManager
00007ff97e84ba78 4000fc8 40 ...ding.WaitCallback 0 instance 0000006b4448d120 _resumeStepsWaitCallback
00007ff97e84f2e8 4000fc9 48 System.EventArgs 0 instance 00000067c44c5fd8 _appEvent
00007ff97e843c08 4000fca 50 ...ections.Hashtable 0 instance 0000000000000000 _handlerFactories
00007ff97e8562d0 4000fcb 58 ...ections.ArrayList 0 instance 0000000000000000 _handlerRecycleList
00007ff97e83e3e0 4000fcc d0 System.Boolean 1 instance 0 _hideRequestResponse
00007ff97bcf4a98 4000fcd 60 ...m.Web.HttpContext 0 instance 00000067c515b380 _context
00007ff97e83fe78 4000fce 68 System.Exception 0 instance 0000000000000000 _lastError
00007ff97e83e3e0 4000fcf d1 System.Boolean 1 instance 1 _timeoutManagerInitialized
00007ff97bcf6e70 4000fd0 70 ....HttpSessionState 0 instance 0000000000000000 _session
00007ff97e844248 4000fd1 78 ...ation.CultureInfo 0 instance 0000000000000000 _appLevelCulture
00007ff97e844248 4000fd2 80 ...ation.CultureInfo 0 instance 0000000000000000 _appLevelUICulture
00007ff97e844248 4000fd3 88 ...ation.CultureInfo 0 instance 0000000000000000 _savedAppLevelCulture
00007ff97e844248 4000fd4 90 ...ation.CultureInfo 0 instance 0000000000000000 _savedAppLevelUICulture
00007ff97e83e3e0 4000fd5 d2 System.Boolean 1 instance 0 _appLevelAutoCulture
00007ff97e83e3e0 4000fd6 d3 System.Boolean 1 instance 0 _appLevelAutoUICulture
00007ff97b8c2930 4000fd7 98 ...ion, System.Web]] 0 instance 0000006b4448cc28 _pipelineEventMasks
00007ff97d68a598 4000fd8 a0 ...ponentModel.ISite 0 instance 0000000000000000 _site
00007ff97e83e3e0 4000fdb d4 System.Boolean 1 instance 1 _initInternalCompleted
00007ff97bcf7040 4000fdc c8 System.Int32 1 instance 0 _appRequestNotifications
00007ff97bcf7040 4000fdd cc System.Int32 1 instance 0 _appPostNotifications
00007ff97e83fc48 4000fde a8 System.String 0 instance 00000068c3697978 _currentModuleCollectionKey
00007ff97e8402c0 4000fe0 b0 System.Object[] 0 instance 0000006b4448a5c0 _moduleContainers
00007ff97e845740 4000fe1 b8 System.Byte[] 0 instance 0000000000000000 _entityBuffer
As we can see, HttpApplication._handlerFactories is null (0000000000000000), I don't understand why this is happening, because from the source code, this field is initialized before constructor calling, and I can find none write operation to it besides this.
So my question is, why is it null?
I have the dump file in case any more information is required.
