4

Currently I am analyzing a dump with WinDbg.

I ran following commands (following Tess' incredible walkthrough):

~* e !clrstack

Which listed me all stacks of all threads. There are 300 running threads with more or less the same stack, so I am just printing the stack of one here ...

OS Thread Id: 0x107c (166)
Child SP       IP Call Site
2bc1e654 77c1015d [HelperMethodFrame_1OBJ: 2bc1e654] System.Threading.WaitHandle.WaitOneNative(System.Runtime.InteropServices.SafeHandle, UInt32, Boolean, Boolean)
2bc1e720 6b2e6dd2 System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean)
2bc1e73c 6b2e6d9c System.Threading.WaitHandle.WaitOne(Int32, Boolean)
2bc1e750 727f4baa System.Threading.ReaderWriterLockSlim.WaitOnEvent(System.Threading.EventWaitHandle, UInt32 ByRef, TimeoutTracker)
2bc1e78c 729bc154 System.Threading.ReaderWriterLockSlim.TryEnterUpgradeableReadLockCore(TimeoutTracker)
2bc1e7b8 725d250c System.Threading.ReaderWriterLockSlim.TryEnterUpgradeableReadLock(TimeoutTracker)
** MORE LINES **

Then I did the following:

> ~166s
eax=00000000 ebx=2bc1e444 ecx=00000000 edx=00000000 esi=00000001 edi=00000000
eip=77c1015d esp=2bc1e3f4 ebp=2bc1e490 iopl=0         nv up ei pl zr na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246
ntdll!NtWaitForMultipleObjects+0x15:
77c1015d 83c404          add     esp,4

Then I was looking for all the instances in this thread:

> !dso
OS Thread Id: 0x107c (166)
ESP/REG  Object   Name
** MORE LINES **
2BC1E7D0 039ec48c System.Threading.ReaderWriterLockSlim
** MORE LINES **

Then I tried to get more information on the ReaderWriterLockSlim-instance:

> !do 039ec48c
Name:        System.Threading.ReaderWriterLockSlim
MethodTable: 725ebda4
EEClass:     724543bc
Size:        68(0x44) bytes
File:        C:\Windows\Microsoft.Net\assembly\GAC_MSIL\System.Core\v4.0_4.0.0.0__b77a5c561934e089\System.Core.dll
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
6b3b8138  4000755       3c       System.Boolean  1 instance        0 fIsReentrant
6b3c3aa4  4000756       1c         System.Int32  1 instance        0 myLock
6b3c7ae4  4000757       20        System.UInt32  1 instance        0 numWriteWaiters
6b3c7ae4  4000758       24        System.UInt32  1 instance        0 numReadWaiters
6b3c7ae4  4000759       28        System.UInt32  1 instance        0 numWriteUpgradeWaiters
6b3c7ae4  400075a       2c        System.UInt32  1 instance      348 numUpgradeWaiters
6b3b8138  400075b       3d       System.Boolean  1 instance        0 fNoWaiters
6b3c3aa4  400075c       30         System.Int32  1 instance      366 upgradeLockOwnerId
6b3c3aa4  400075d       34         System.Int32  1 instance      366 writeLockOwnerId
6b3c0ac0  400075e        c ...g.EventWaitHandle  0 instance 00000000 writeEvent
6b3c0ac0  400075f       10 ...g.EventWaitHandle  0 instance 00000000 readEvent
6b3c0ac0  4000760       14 ...g.EventWaitHandle  0 instance 08188858 upgradeEvent
6b3c0ac0  4000761       18 ...g.EventWaitHandle  0 instance 00000000 waitUpgradeEvent
6b3b821c  4000763        4         System.Int64  1 instance 231 lockID
6b3b8138  4000765       3e       System.Boolean  1 instance        0 fUpgradeThreadHoldingRead
6b3c7ae4  4000766       38        System.UInt32  1 instance 2147483649 owners
6b3b8138  4000767       3f       System.Boolean  1 instance        0 fDisposed
6b3b821c  4000762      3e0         System.Int64  1   shared   static s_nextLockID
    >> Domain:Value  01742528:NotInit  01783fb8:NotInit  1268c9d8:NotInit  <<
725fd46c  4000764        4 ...ReaderWriterCount  0   shared TLstatic t_rwc
    >> Thread:Value <<

I've matched the value of owners (2147483649) against the the information on this page = 0x80000001

private const uint WRITER_HELD = 0x80000000;
private const uint WAITING_WRITERS = 0x40000000; 
private const uint WAITING_UPGRADER = 0x20000000;

But how can I identify the thread which holds the lock?

2 Answers2

3

If you have a deadlock, you can use SOSEx's !dlk command which does all the work for you.

*DEADLOCK DETECTED*
CLR thread 0x4 holds the Writer lock on ReaderWriterLockSlim 02712580
...and is waiting for a Reader lock on ReaderWriterLockSlim 0271253c
CLR thread 0x3 holds the Writer lock on ReaderWriterLockSlim 0271253c
...and is waiting for a Reader lock on ReaderWriterLockSlim 02712580
CLR Thread 0x4 is waiting at System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean)(+0x1f IL,+0x22 Native)
CLR Thread 0x3 is waiting at System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean)(+0x1f IL,+0x22 Native)

If you don't have a deadlock, you can still use SOSEx, but type !mlocks. The output looks like this, so you get all the different thread numbers and the type of the lock.

0:012> !mlocks
Examining SyncBlocks...
Scanning for ReaderWriterLock instances...
Scanning for holders of ReaderWriterLock locks...
Scanning for ReaderWriterLockSlim instances...
Scanning for holders of ReaderWriterLockSlim locks...
Examining CriticalSections...

ClrThread  DbgThread  OsThread    LockType    Lock      LockLevel
----------------------------------------------------------------------
0x1        0          0x1460      thinlock    02718bcc  (recursion:0)
0x5        6          0x1e80      RWLock      027125f0  Writer        
0x8        9          0x22ac      CritSect    027124c0                
0x9        10         0x27b8      SyncBlock   0045f4e8                
0xa        11         0x33f8      SyncBlock   0045f4b4                
0x7        8          0x388c      CritSect    027124a0                
0x4        5          0x3d20      RWLockSlim  02712580  Writer        
0x3        4          0x3e44      RWLockSlim  0271253c  Writer        
0x6        7          0x4704      RWLock      027125c4  Writer

If you have DML enabled, you can even click the links in the Lock column which gives more information. You can also type !rwlock <lock> if you don't like DML.

Thomas Weller
  • 55,411
  • 20
  • 125
  • 222
  • +, hehe ... cool stuff! the only thing is ... I am getting around 11 **thousand** lines as output. Way too much to be easily interpreted - but when I search for `RwLockSlim`, I am getting the needed line... –  Feb 26 '14 at 13:39
  • 2
    @AndreasNiedermair: `.shell -ci "!mlocks -d" find "Writer"` may help if the output for readers is "Reader" (I can't try at the moment) – Thomas Weller Feb 26 '14 at 14:49
2

I did some more research, and decompiled ReaderWriterLockSlim and determined the meaning of writeLockOwnerId and upgradeLockOwnerId: They are actually the managed thread ids of the threads within the corresponding method.

Unfortunately you have to determine the ordinal of this managed thread id, by doing:

> !threads
ThreadCount:      371
UnstartedThread:  0
BackgroundThread: 371
PendingThread:    0
DeadThread:       0
Hosted Runtime:   no
                                                                         Lock  
       ID OSID ThreadOBJ    State GC Mode     GC Alloc Context  Domain   Count Apt Exception
** MORE LINES **
 381  366 1c0c 1e1fca78   1029220 Preemptive  3E46102C:00000000 01783fb8 2     MTA (Threadpool Worker) 
** MORE LINES **

After this, you can simply do:

> ~381s
eax=00000000 ebx=00000000 ecx=00000000 edx=00000000 esi=0000432d edi=4519d490
eip=77c0f8d1 esp=4519d448 ebp=4519d4b4 iopl=0         nv up ei pl zr na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246
ntdll!ZwWaitForSingleObject+0x15:
77c0f8d1 83c404          add     esp,4

> !clrstack
OS Thread Id: 0x1c0c (381)
Child SP       IP Call Site
** MORE LINES **
4519da08 69f35592 System.Data.SqlClient.SqlCommand.ExecuteReader(System.Data.CommandBehavior, System.String)
 ** MORE LINES **

Et voilà: the thread, which holds the upgradable lock, is currently executing some sql.