java ReentrantReadWriteLock "max lock count exceeded" troubleshooting

Question

I'm debugging an error from the application on a ReadWriteLock, as shown below in the stack traces. For the particular ReentrantReadWriteLock, you can imagine there's a global lock in an RPC server. All RPC worker threads will acquire a read lock to serve, and one checkpointing thread will acquire the write lock to take a checkpoint.

java.lang.Error: Maximum lock count exceeded
  at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.fullTryAcquireShared(ReentrantReadWriteLock.java:528)
  at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryAcquireShared(ReentrantReadWriteLock.java:488)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
  at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lockInterruptibly(ReentrantReadWriteLock.java:772)
  ...some application call stack acquiring a ReadWriteLock.readLock()

From the stack trace, I'm only able to tell the lock count is on the ReadLock(not the WriteLock). From the java code, the "max count" is exceeded when it is 65535. My questions are:

Does that mean 65535 read lock holders? Or one thread holding the thread with 65535 recursions? Or a combination of both? How do I find out which thread is doing the heavy recursion?

I'm almost certain that there are around 500 RPC worker threads serving(not 65535 threads), therefore some threads must be acquiring the lock more than 1 time, a recursive operation somewhere essentially doing the logic below:

// pseudo code
public void serveRPC() {
  // Always using try-with-resource so I assume all locks are released properly
  try (acquireSharedLock(stateLock)) {
    // something calling serveRPC so recursively acquiring the lock in the same thread!
  }
}

There are some options I can think of, what are your suggestions? Thanks!

// Option 1: check the holder count on every lock acquisition
public AutoCloseable<Lock> acquireStateLockShared() {
  // Question: is this number accurate?
  // Question: does lock.getReadLockCount() also matter?
  if (stateLock.getReadHoldCount() > 1) { // print something }
  ...
}

// Option 2: Have something like a global bookkeeper to record the usage of each thread
Map<Long, LongAdder> recordTable = ...
public AutoCloseable<Lock> acquireStateLockShared() {
    recordTable.computeIfAbsent(Thread.currentThread().getId(), k -> new AtomicInteger(0)).increment()
    ..
}
// when the AutoCloseable is closed, reduce in the recordTable

java ReentrantReadWriteLock "max lock count exceeded" troubleshooting

0 Answers0