4

I have a thread dump and Memory dump of my process where a thread is hung

at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000070feebf40> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
at voldemort.store.readonly.chunk.DataFileChunkSetIterator.<init>(DataFileChunkSetIterator.java:66)
at voldemort.store.readonly.chunk.ChunkedFileSet$ROKeyIterator.<init>(ChunkedFileSet.java:617)
at voldemort.store.readonly.ReadOnlyStorageEngine.keys(ReadOnlyStorageEngine.java:478)
at voldemort.server.protocol.admin.FullScanFetchStreamRequestHandler.<init>(FullScanFetchStreamRequestHandler.java:66)

at voldemort.server.protocol.admin.FullScanFetchEntriesRequestHandler.(FullScanFetchEntriesRequestHandler.java:53)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

But I need to see what object is at the memory location 0x000000070feebf40 . Is there an easy way to see this object ? Both YourKit and jvisualvm does not see what is in at this memory address.

I tried looking by instances but there are 300 of them for most of them which makes it really difficult to analyze.

I have used C# and there you can easily search the memory using WinDbg and sos plugin.

Arun Thirupathi
  • 351
  • 2
  • 11

3 Answers3

6

If you have the Heap Dump and the location of the object you're trying to find, load that dump in VisualVM and use the following Query in OQL Console

select heap.findObject("0x00000000fe9d4910")
Arkantos
  • 6,530
  • 2
  • 16
  • 36
0

I do not think that 0x000000070feebf40 is an address in memory. This is object identity hash code. Identity hash-code is persistent, but address in memory is volatile (GC moves objects).

Vladimir Kondratyev
  • 709
  • 5
  • 6
  • GC Moving objects is a good point. But the address is real address and not the hash code. To be sure, in the future, I will take a thread dump before and after the memory dump to ensure that the object is not moved. – Arun Thirupathi Feb 17 '15 at 17:57
0

And what exactly would you get after figuring out the object in memory? That's an internal object of ReentrantReadWriteLock at which the thread had parked waiting for acquire to succeed (which means the lock is busy). To diagnose just about any real issue, it is way better to look into the source. This part of the stack:

...
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
at voldemort.store.readonly.chunk.DataFileChunkSetIterator.<init>(DataFileChunkSetIterator.java:66)
...

...means you are looking for a ReentrantReadWriteLock.readLock().lock() in DataFileChunkSetIterator constructor.

Aleksey Shipilev
  • 18,599
  • 2
  • 67
  • 86
  • 1
    Thanks for the comment. I have more than 400 instances of ReentrantReadWriteLock , I need to find for which instance the threads are competing for. Once I got that, I can look at the object that is holding on the reference to this object and further up the chain and figure out what is going on. – Arun Thirupathi Feb 17 '15 at 17:55