First I'll summarize what I've found so far.
- This answer suggests that changing the
concurrencyLevel
parameter ofConcurrentHashMap
's constructor might help. I've tried that and my code still hanged. - Answers here suggest that it could be a runtime bug.
What I'm trying to do:
- I have 10 worker threads running along with a main thread. The worker threads will have to process many arrays to find the index of the max element in the array (if there are multiple max values, the first occurrence will be used). Among these "many arrays," some of them can be duplicate, so I'm trying to avoid those full array scans to speed up the program.
- The controller class contains a
ConcurrentHashMap
that maps the hash values of arrays to the corresponding max-element indices. - The worker threads will ask the controller class for the mapped index first before trying to calculate the index by doing full array scans. In the latter case, the newly calculated index will be put into the map.
- The main thread does not access the hash map.
What happened:
- My code will hang after 70,000 ~ 130,000 calls to
getMaxIndex()
. This count is obtained by putting a log string intogetMaxIndex()
so it might not be exactly accurate. - My CPU usage will gradually go up for ~6 seconds, and then it will go down to ~10% after peaked at ~100%. I have plenty of unused memory left. (Does this look like deadlock?)
- If the code does not use map it works just fine (see
getMaxIndex()
version 2 below). - I've tried to add
synchronized
togetMaxIndex()
's signature and use the regularHashMap
instead, that also did not work. - I've tried to use different
initialCapacity
values too (e.g. 50,000 & 100,000). Did not work.
Here's my code:
// in the controller class
int getMaxIndex(@NotNull double[] arr) {
int hash = Arrays.hashCode(arr);
if(maxIndices.containsKey(hash)) {
return maxIndices.get(hash);
} else {
int maxIndex =
IntStream.range(0, arr.length)
.reduce((a, b) -> arr[a] < arr[b] ? b : a)
.orElse(-1); // -1 to let program crash
maxIndices.put(hash, maxIndex);
return maxIndex;
}
}
The worker thread will call getMaxIndex()
like this: return remaining[controller.getMaxIndex(arr)];
, remaining
is just another int
array.
getMaxIndex()
v2:
int getMaxIndex(@NotNull double[] arr) {
return IntStream.range(0, arr.length)
.reduce((a, b) -> arr[a] < arr[b] ? b : a)
.orElse(-1); // -1 to let program crash
}
JVM info in case it matters:
java version "1.8.0_151"
Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)
EDIT: stack dump; I used Phaser
to synchronize the worker threads, so some of them appear to be waiting on the phaser, but pool-1-thread-2
, pool-1-thread-10
, pool-1-thread-11
, and pool-1-thread-12
do not appear to be waiting on the phaser.
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.151-b12 mixed mode):
"Attach Listener" #23 daemon prio=9 os_prio=0 tid=0x00007f0c54001000 nid=0x4da2 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"pool-1-thread-13" #22 prio=5 os_prio=0 tid=0x00007f0c8c2cb800 nid=0x4d5e waiting on condition [0x00007f0c4eddd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000076e792f40> (a java.util.concurrent.Phaser$QNode)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.Phaser$QNode.block(Phaser.java:1140)
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
at java.util.concurrent.Phaser.internalAwaitAdvance(Phaser.java:1067)
at java.util.concurrent.Phaser.arriveAndAwaitAdvance(Phaser.java:690)
at Ant.call(Ant.java:77)
at Ant.call(Ant.java:10)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"pool-1-thread-12" #21 prio=5 os_prio=0 tid=0x00007f0c8c2ca000 nid=0x4d5d waiting on condition [0x00007f0c4eede000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000775518738> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"pool-1-thread-11" #20 prio=5 os_prio=0 tid=0x00007f0c8c2c8000 nid=0x4d5c waiting on condition [0x00007f0c4efdf000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000775518738> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"pool-1-thread-10" #19 prio=5 os_prio=0 tid=0x00007f0c8c2c6000 nid=0x4d5b waiting on condition [0x00007f0c4f0e0000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000775518738> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"pool-1-thread-9" #18 prio=5 os_prio=0 tid=0x00007f0c8c2c4800 nid=0x4d5a waiting on condition [0x00007f0c4f1e1000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000076e7c74f8> (a java.util.concurrent.Phaser$QNode)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.Phaser$QNode.block(Phaser.java:1140)
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
at java.util.concurrent.Phaser.internalAwaitAdvance(Phaser.java:1067)
at java.util.concurrent.Phaser.arriveAndAwaitAdvance(Phaser.java:690)
at Ant.call(Ant.java:77)
at Ant.call(Ant.java:10)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"pool-1-thread-8" #17 prio=5 os_prio=0 tid=0x00007f0c8c2c2800 nid=0x4d59 waiting on condition [0x00007f0c4f2e2000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000076e64fb78> (a java.util.concurrent.Phaser$QNode)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.Phaser$QNode.block(Phaser.java:1140)
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
at java.util.concurrent.Phaser.internalAwaitAdvance(Phaser.java:1067)
at java.util.concurrent.Phaser.arriveAndAwaitAdvance(Phaser.java:690)
at Ant.call(Ant.java:77)
at Ant.call(Ant.java:10)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"pool-1-thread-7" #16 prio=5 os_prio=0 tid=0x00007f0c8c2c1000 nid=0x4d58 waiting on condition [0x00007f0c4f3e3000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000076e8b44c8> (a java.util.concurrent.Phaser$QNode)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.Phaser$QNode.block(Phaser.java:1140)
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
at java.util.concurrent.Phaser.internalAwaitAdvance(Phaser.java:1067)
at java.util.concurrent.Phaser.arriveAndAwaitAdvance(Phaser.java:690)
at Ant.call(Ant.java:77)
at Ant.call(Ant.java:10)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"pool-1-thread-6" #15 prio=5 os_prio=0 tid=0x00007f0c8c2bf800 nid=0x4d57 waiting on condition [0x00007f0c4f4e4000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000076e5b4500> (a java.util.concurrent.Phaser$QNode)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.Phaser$QNode.block(Phaser.java:1140)
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
at java.util.concurrent.Phaser.internalAwaitAdvance(Phaser.java:1067)
at java.util.concurrent.Phaser.arriveAndAwaitAdvance(Phaser.java:690)
at Ant.call(Ant.java:77)
at Ant.call(Ant.java:10)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"pool-1-thread-5" #14 prio=5 os_prio=0 tid=0x00007f0c8c2bd800 nid=0x4d56 waiting on condition [0x00007f0c4f5e5000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000076e836958> (a java.util.concurrent.Phaser$QNode)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.Phaser$QNode.block(Phaser.java:1140)
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
at java.util.concurrent.Phaser.internalAwaitAdvance(Phaser.java:1067)
at java.util.concurrent.Phaser.arriveAndAwaitAdvance(Phaser.java:690)
at Ant.call(Ant.java:77)
at Ant.call(Ant.java:10)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"pool-1-thread-4" #13 prio=5 os_prio=0 tid=0x00007f0c8c2bc000 nid=0x4d55 waiting on condition [0x00007f0c4f6e6000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000076e4f4cf0> (a java.util.concurrent.Phaser$QNode)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.Phaser$QNode.block(Phaser.java:1140)
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
at java.util.concurrent.Phaser.internalAwaitAdvance(Phaser.java:1067)
at java.util.concurrent.Phaser.arriveAndAwaitAdvance(Phaser.java:690)
at Ant.call(Ant.java:77)
at Ant.call(Ant.java:10)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"pool-1-thread-3" #12 prio=5 os_prio=0 tid=0x00007f0c8c2ba000 nid=0x4d54 waiting on condition [0x00007f0c4f7e7000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000076e40abb8> (a java.util.concurrent.Phaser$QNode)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.Phaser$QNode.block(Phaser.java:1140)
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
at java.util.concurrent.Phaser.internalAwaitAdvance(Phaser.java:1067)
at java.util.concurrent.Phaser.arriveAndAwaitAdvance(Phaser.java:690)
at Ant.call(Ant.java:77)
at Ant.call(Ant.java:10)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"pool-1-thread-2" #11 prio=5 os_prio=0 tid=0x00007f0c8c2b8800 nid=0x4d53 waiting on condition [0x00007f0c4f8e8000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000775518738> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"pool-1-thread-1" #10 prio=5 os_prio=0 tid=0x00007f0c8c2b5800 nid=0x4d52 waiting on condition [0x00007f0c4f9e9000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000076e486ab0> (a java.util.concurrent.Phaser$QNode)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.Phaser$QNode.block(Phaser.java:1140)
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
at java.util.concurrent.Phaser.internalAwaitAdvance(Phaser.java:1067)
at java.util.concurrent.Phaser.arriveAndAwaitAdvance(Phaser.java:690)
at Ant.call(Ant.java:77)
at Ant.call(Ant.java:10)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f0c8c200800 nid=0x4d50 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f0c8c1fd800 nid=0x4d4f waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f0c8c1f8800 nid=0x4d4e waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f0c8c1f7800 nid=0x4d4d waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Monitor Ctrl-Break" #5 daemon prio=5 os_prio=0 tid=0x00007f0c8c1fb000 nid=0x4d4c runnable [0x00007f0c781b4000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
- locked <0x000000077550ecb0> (a java.io.InputStreamReader)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:161)
at java.io.BufferedReader.readLine(BufferedReader.java:324)
- locked <0x000000077550ecb0> (a java.io.InputStreamReader)
at java.io.BufferedReader.readLine(BufferedReader.java:389)
at com.intellij.rt.execution.application.AppMainV2$1.run(AppMainV2.java:64)
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f0c8c181000 nid=0x4d49 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f0c8c14d800 nid=0x4d42 in Object.wait() [0x00007f0c78564000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000775500d08> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x0000000775500d08> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f0c8c149000 nid=0x4d41 in Object.wait() [0x00007f0c78665000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000775500d48> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x0000000775500d48> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
"main" #1 prio=5 os_prio=0 tid=0x00007f0c8c00c800 nid=0x4d35 waiting on condition [0x00007f0c91f77000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000076dd5e268> (a java.util.concurrent.FutureTask)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
at java.util.concurrent.FutureTask.get(FutureTask.java:191)
at java.util.concurrent.AbstractExecutorService.invokeAll(AbstractExecutorService.java:244)
at ConcurrentACS.loop(ConcurrentACS.java:138)
at ConcurrentACS.compute(ConcurrentACS.java:165)
at ConcurrentACS.main(ConcurrentACS.java:192)
"VM Thread" os_prio=0 tid=0x00007f0c8c141800 nid=0x4d3f runnable
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f0c8c022000 nid=0x4d37 runnable
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f0c8c024000 nid=0x4d38 runnable
"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f0c8c025800 nid=0x4d39 runnable
"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f0c8c027800 nid=0x4d3a runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00007f0c8c205800 nid=0x4d51 waiting on condition
JNI global references: 272