How to find the HttpWebRequest/Response or Connections blocked by the wrong setting of the limitation of ServicePoint using .net debugging tools

Question

Today, I have an issue, my production web server has very high iis connections, the cpu and memory is on normal level. From Zabbix(an open source monitoring software, collecting data from server/performance counter, displaying on web), I can see a lot of "ASP.NET Requests Queued (a performance couter)". so I get a dump from the server. I use DebugDiag2.2 to automatically analyze the dump. It reports that:

This thread is making an HttpWebRequest and waiting on the remote server to respond. One or more of these requests are using at least half of its maximum number of available connections.

Warning, at least half of the availabe connections are being used

HttpRequest URI:http://ws.product.aaa.bbb.com/product.search/api/x-protobuf/SearchData ServicePoint - ConnectionLimit:32 CurrentConnections:31

I google for a while, then find the link "asp-net-performance-case-study-web-service-calls-taking-forever" and "Timeouts When Making Web Requests in .NET"

My server uses .net 4, I try to follow the steps to dump out the waitList, in which are those iis connections drawn on the Zabbix.

0:048> !dumpheap -type System.Net.ConnectStream
         Address               MT     Size
000000d2395e0fe8 00007ffbc6052c70      176     
000000d2395e1148 00007ffbc6055440       24     
000000d2395f5780 00007ffbc6052c70      176  



/*dump the ConnectStream to find m_connection   */
0:048> !do 000000d2395e0fe8
Name:        System.Net.ConnectStream
MethodTable: 00007ffbc6052c70
EEClass:     00007ffbc5d29f78
Size:        176(0xb0) bytes
File:        C:\Windows\Microsoft.Net\assembly\GAC_MSIL\System\v4.0_4.0.0.0__b77a5c561934e089\System.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
...
00007ffbc6f737c8  4001d16       78         System.Int32  1 instance                3 m_CallNesting
00007ffbc60649e0  4001d17       20 ...tterGatherBuffers  0 instance 000000d2395e1098 m_BufferedData
00007ffbc6f6f370  4001d18       9c       System.Boolean  1 instance                0 m_SuppressWrite
00007ffbc6f6f370  4001d19       9d       System.Boolean  1 instance                1 m_BufferOnly
00007ffbc6f92ac0  4001d1a       68         System.Int64  1 instance 0 m_BytesLeftToWrite
00007ffbc6f737c8  4001d1b       7c         System.Int32  1 instance                0 m_BytesAlreadyTransferred
00007ffbc6051148  4001d1c       28 ...em.Net.Connection  0 instance 000000d53828cf18 m_Connection
00007ffbc6f766d0  4001d1d       30        System.Byte[]  0 instance 0000000000000000 m_ReadBuffer
00007ffbc6f737c8  4001d1e       80         System.Int32  1 instance                0 m_ReadOffset
00007ffbc6f737c8  4001d1f       84         System.Int32  1 instance                0 m_ReadBufferSize
00007ffbc6f92ac0  4001d20       70         System.Int64  1 instance 0 m_ReadBytes
00007ffbc6f6f370  4001d21       9e       System.Boolean  1 instance                0 m_Chunked
00007ffbc6f737c8  4001d22       88         System.Int32  1 instance                2 m_DoneCalled
00007ffbc6f737c8  4001d23       8c         System.Int32  1 instance                2 m_ShutDown
00007ffbc6f70e08  4001d24       38     System.Exception  0 instance 0000000000000000 m_ErrorException
00007ffbc6f6f370  4001d25       9f       System.Boolean  1 instance                0 m_ChunkEofRecvd
00007ffbc6057300  4001d26       40 ...m.Net.ChunkParser  0 instance 0000000000000000 m_ChunkParser
00007ffbc6071f28  4001d27       90         System.Int32  1 instance                3 m_HttpWriteMode
00007ffbc6f737c8  4001d28       94         System.Int32  1 instance           300000 m_ReadTimeout
00007ffbc6f737c8  4001d29       98         System.Int32  1 instance           300000 m_WriteTimeout
00007ffbc6052760  4001d2a       48 ...estLifetimeSetter  0 instance 0000000000000000 m_RequestLifetimeSetter
00007ffbc6f7c8c8  4001d2b       50 System.AsyncCallback  0 instance 000000d2395e10c8 m_ReadCallbackDelegate
00007ffbc6f7c8c8  4001d2c       58 System.AsyncCallback  0 instance 000000d2395e1108 m_WriteCallbackDelegate
00007ffbc60a4850  4001d2f       60 ...et.HttpWebRequest  0 instance 000000d2395e0338 m_Request
00007ffbc6f6f370  4001d31       a0       System.Boolean  1 instance                0 m_IgnoreSocketErrors
00007ffbc6f6f370  4001d32       a1       System.Boolean  1 instance                0 m_ErrorResponseStatus



/*find m_connection, then to dump the waitlist */
0:048> !do 000000d53828cf18 
Name:        System.Net.Connection
MethodTable: 00007ffbc6051148
EEClass:     00007ffbc5cc1ff8
Size:        296(0x128) bytes
File:        C:\Windows\Microsoft.Net\assembly\GAC_MSIL\System\v4.0_4.0.0.0__b77a5c561934e089\System.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
....
00007ffbc6f6f370  4001c6d       5c       System.Boolean  1 instance                0 m_CheckLifetime
00007ffbc6f87580  4001c6e       60      System.TimeSpan  1 instance 000000d53828cf78 m_Lifetime
00007ffbc6f87af8  4001c6f       68      System.DateTime  1 instance 000000d53828cf80 m_CreateTime
00007ffbc6f6f370  4001c70       5d       System.Boolean  1 instance                0 m_ConnectionIsDoomed
00007ffbc6075a30  4001c71       20 ...et.ConnectionPool  0 instance 0000000000000000 m_ConnectionPool
00007ffbc6f874e0  4001c72       28 System.WeakReference  0 instance 000000d53828d040 m_Owner
00007ffbc6f737c8  4001c73       58         System.Int32  1 instance               -1 m_PooledCount
00007ffbc6f6f370  4001c74       5e       System.Boolean  1 instance                0 m_Initalizing
00007ffbc6099378  4001c75       30 System.Net.IPAddress  0 instance 000000d238ee5040 m_ServerAddress
00007ffbc60546e0  4001c76       38 ...ets.NetworkStream  0 instance 000000d53828d058 m_NetworkStream
00007ffbc609b3c0  4001c77       40 ...et.Sockets.Socket  0 instance 0000000000000000 m_AbortSocket
00007ffbc609b3c0  4001c78       48 ...et.Sockets.Socket  0 instance 0000000000000000 m_AbortSocket6
00007ffbc6f6f370  4001c79       5f       System.Boolean  1 instance                1 m_JustConnected
00007ffbc6053ad0  4001c7a       50 ...eralAsyncDelegate  0 instance 000000d53828d218 m_AsyncCallback
00007ffbc6099bc8  4001ccb       e0         System.Int32  1 instance                0 m_Error
00007ffbc6f70e08  4001ccc       70     System.Exception  0 instance 0000000000000000 m_InnerException
00007ffbc6f737c8  4001ccd       e4         System.Int32  1 instance                7 m_IISVersion
00007ffbc6f766d0  4001cce       78        System.Byte[]  0 instance 000000d5390cca00 m_ReadBuffer
00007ffbc6f6f370  4001ccf      104       System.Boolean  1 instance                1 m_ReadBufferFromPinnableCache
00007ffbc6f737c8  4001cd0       e8         System.Int32  1 instance                0 m_BytesRead
00007ffbc6f737c8  4001cd1       ec         System.Int32  1 instance                0 m_BytesScanned
00007ffbc6f737c8  4001cd2       f0         System.Int32  1 instance              505 m_TotalResponseHeadersLength
00007ffbc6f737c8  4001cd3       f4         System.Int32  1 instance            65536 m_MaximumResponseHeadersLength
00007ffbc6f92ac0  4001cd4       d8         System.Int64  1 instance -1 m_MaximumUnauthorizedUploadLength
00007ffbc6053fc8  4001cd5       80 ....CoreResponseData  0 instance 000000d239a4a930 m_ResponseData
00007ffbc605fd38  4001cd6       f8         System.Int32  1 instance                0 m_ReadState
00007ffbc6056670  4001cd7       88 ...+StatusLineValues  0 instance 000000d53828d168 m_StatusLineValues
00007ffbc6f737c8  4001cd8       fc         System.Int32  1 instance                5 m_StatusState
00007ffbc5c8b0c8  4001cd9       90 ...istItem, System]]  0 instance 000000d53828d098 m_WaitList
00007ffbc6f87260  4001cda       98 ...ections.ArrayList  0 instance 000000d53828d0c0 m_WriteList
00007ffbc6f81430  4001cdb       a0  System.IAsyncResult  0 instance 0000000000000000 m_LastAsyncResult
00007ffbc6063a50  4001cdc       a8 ...TimerThread+Timer  0 instance 000000d53828d190 m_RecycleTimer
00007ffbc607e258  4001cdd      110 ...Net.WebParseError  1 instance 000000d53828d028 m_ParseError
00007ffbc6f6f370  4001cde      105       System.Boolean  1 instance                1 m_AtLeastOneResponseReceived
00007ffbc6056328  4001ce3       b0 ...HttpAbortDelegate  0 instance 000000d53828d0e8 m_AbortDelegate
00007ffbc6056100  4001ce4       b8 ...t.ConnectionGroup  0 instance 000000d2384cdc18 m_ConnectionGroup
00007ffbc6056f58  4001ce5       c0 ...onnectionDelegate  0 instance 000000d53828d128 m_ConnectionUnlock
00007ffbc6f87af8  4001ce6      118      System.DateTime  1 instance 000000d53828d030 m_IdleSinceUtc
00007ffbc60a4850  4001ce7       c8 ...et.HttpWebRequest  0 instance 0000000000000000 m_LockedRequest
00007ffbc60a4850  4001ce8       d0 ...et.HttpWebRequest  0 instance 0000000000000000 m_CurrentRequest
00007ffbc6f6f370  4001ce9      106       System.Boolean  1 instance                1 m_CanPipeline
00007ffbc6f6f370  4001cea      107       System.Boolean  1 instance                1 m_Free
00007ffbc6f6f370  4001ceb      108       System.Boolean  1 instance                1 m_Idle
00007ffbc6f6f370  4001cec      109       System.Boolean  1 instance                1 m_KeepAlive
00007ffbc6f6f370  4001ced      10a       System.Boolean  1 instance                0 m_Pipelining
00007ffbc6f737c8  4001cee      100         System.Int32  1 instance                0 m_ReservedCount
00007ffbc6f6f370  4001cef      10b       System.Boolean  1 instance                1 m_ReadDone
00007ffbc6f6f370  4001cf0      10c       System.Boolean  1 instance                1 m_WriteDone
00007ffbc6f6f370  4001cf1      10d       System.Boolean  1 instance                0 m_RemovedFromConnectionList
00007ffbc6f6f370  4001cf2      10e       System.Boolean  1 instance                0 m_NonKeepAliveRequestPipelined
00007ffbc6f6f370  4001cf3      10f       System.Boolean  1 instance                0 m_IsPipelinePaused

I find the wailist, but the size is 0.

0:048> !do 000000d53828d098
Name:        System.Collections.Generic.List`1[[System.Net.Connection+WaitListItem, System]]
MethodTable: 00007ffbc5c8b0c8
EEClass:     00007ffbc69b8c48
Size:        40(0x28) bytes
File:        C:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
00007ffbc6f71250  4000cd1        8      System.Object[]  0 instance 000000d43805cb70 _items
00007ffbc6f737c8  4000cd2       18         System.Int32  1 instance                0 _size
00007ffbc6f737c8  4000cd3       1c         System.Int32  1 instance                0 _version
00007ffbc6f711b8  4000cd4       10        System.Object  0 instance 0000000000000000 _syncRoot

I use a foreach loop to dump all the waitlist,

.foreach ( obj {!dumpheap -mt 00007ffbc6051148 -short}){.echo ******************; !do poi(${obj}+0x90) }
....
              MT    Field   Offset                 Type VT     Attr            Value Name
00007ffbc6f71250  4000cd1        8      System.Object[]  0 instance 000000d43805cb70 _items
00007ffbc6f737c8  4000cd2       18         System.Int32  1 instance                0 _size

....

in the output, _size=0 and _items =000000d43805cb70.

as of now, I have no idea to find out where the "Queued Reqeusts" are. Could you please help to find out?

I can't follow your debug session, perhaps because I don't know what a Zabbix is. It's unclear to me where you get all the object addresses from. Since I can't help much, I only have one recommendation: have a look at https://netext.codeplex.com/ which is a WinDbg plugin focusing on HTTP and network analysis. Perhaps you can benefit from one of the `!from ... where ... select` statements to query all objects on the heap — Thomas Weller, Sep 20 '16 at 17:13
Thanks, I didn't post much details, because I think the blog https://blogs.msdn.microsoft.com/tess/2006/02/23/asp-net-performance-case-study-web-service-calls-taking-forever/ gives very detailed info. I just follow these steps, the only difference is my local object address and the offset of object's proerty. — gfan, Sep 21 '16 at 02:53

How to find the HttpWebRequest/Response or Connections blocked by the wrong setting of the limitation of ServicePoint using .net debugging tools

0 Answers0