022-12-20 02:09:13.085 [Gateway Sender Primary Lock Acquisition Thread Volunteer] INFO org.apache.geode.internal.cache.wan.GatewaySenderAdvisor - org.apache.geode.internal.cache.wan.GatewaySenderAdvisor@31e1b6b8 is becoming primary gateway Sender.
2022-12-20 02:09:13.085 [Gateway Sender Primary Lock Acquisition Thread Volunteer] INFO org.apache.geode.internal.cache.wan.GatewaySenderAdvisor - SerialGatewaySender{id=sender1,remoteDsId=2,isRunning =true,isPrimary =false} : Starting as primary
2022-12-20 02:09:13.128 [Gateway Sender Primary Lock Acquisition Thread Volunteer] INFO org.apache.geode.internal.cache.wan.GatewaySenderAdvisor - SerialGatewaySender{id=sender1,remoteDsId=2,isRunning =true,isPrimary =false} : Becoming primary gateway sender
2022-12-20 02:09:13.129 [Event Processor for GatewaySender_sender1.3] INFO org.apache.geode.internal.cache.wan.serial.SerialGatewaySenderEventProcessor - Gateway Failover Initiated: Processing 5 unprocessed events.
2022-12-20 02:09:13.129 [Event Processor for GatewaySender_sender1.2] INFO org.apache.geode.internal.cache.wan.serial.SerialGatewaySenderEventProcessor - Gateway Failover Initiated: Processing 6 unprocessed events.
2022-12-20 02:09:13.129 [Event Processor for GatewaySender_sender1.1] INFO org.apache.geode.internal.cache.wan.serial.SerialGatewaySenderEventProcessor - Gateway Failover Initiated: Processing 2 unprocessed events.
2022-12-20 02:09:13.129 [Event Processor for GatewaySender_sender1.4] INFO org.apache.geode.internal.cache.wan.serial.SerialGatewaySenderEventProcessor - Gateway Failover Initiated: Processing 6 unprocessed events.
2022-12-20 02:09:13.129 [Event Processor for GatewaySender_sender1.0] INFO org.apache.geode.internal.cache.wan.serial.SerialGatewaySenderEventProcessor - Gateway Failover Initiated: Processing 6 unprocessed events.
2022-12-20 02:09:13.143 [Pooled Serial Message Processor2-1] INFO org.apache.geode.internal.cache.wan.serial.SerialGatewaySenderEventProcessor - GatewayEventProcessor[gatewaySenderId=sender1;remoteDSId=2;batchSize=100] : Waiting for failover completion
2022-12-20 02:09:13.146 [Event Processor for GatewaySender_sender1.1] INFO org.apache.geode.internal.cache.wan.serial.SerialGatewaySenderEventProcessor - SerialGatewaySender{id=sender1,remoteDsId=2,isRunning =true,isPrimary =true} : Marking 5 events as possible duplicates
2022-12-20 02:09:13.148 [Event Processor for GatewaySender_sender1.3] INFO org.apache.geode.internal.cache.wan.serial.SerialGatewaySenderEventProcessor - SerialGatewaySender{id=sender1,remoteDsId=2,isRunning =true,isPrimary =true} : Marking 6 events as possible duplicates
2022-12-20 02:09:13.149 [Event Processor for GatewaySender_sender1.4] INFO org.apache.geode.internal.cache.wan.serial.SerialGatewaySenderEventProcessor - SerialGatewaySender{id=sender1,remoteDsId=2,isRunning =true,isPrimary =true} : Marking 7 events as possible duplicates
2022-12-20 02:09:13.150 [Event Processor for GatewaySender_sender1.2] INFO org.apache.geode.internal.cache.wan.serial.SerialGatewaySenderEventProcessor - SerialGatewaySender{id=sender1,remoteDsId=2,isRunning =true,isPrimary =true} : Marking 9 events as possible duplicates
2022-12-20 02:09:13.150 [Event Processor for GatewaySender_sender1.0] INFO org.apache.geode.internal.cache.wan.serial.SerialGatewaySenderEventProcessor - SerialGatewaySender{id=sender1,remoteDsId=2,isRunning =true,isPrimary =true} : Marking 9 events as possible duplicates
2022-12-20 02:09:17.146 [Pooled High Priority Message Processor 15] INFO org.apache.geode.distributed.internal.membership.gms.Services - received leave request from 10.4.20.34(20.34-S1:121257)<v203>:1025 for 10.4.20.34(20.34-S1:121257)<v203>:1025
2022-12-20 02:09:17.147 [Pooled High Priority Message Processor 15] INFO org.apache.geode.distributed.internal.membership.gms.Services - JoinLeave.processMessage(LeaveRequestMessage) invoked. isCoordinator=false; isStopping=false; cancelInProgress=false
2022-12-20 02:09:17.147 [Pooled High Priority Message Processor 15] INFO org.apache.geode.distributed.internal.membership.gms.Services - Checking to see if I should become coordinator. My address is 10.4.20.148(20.148-S1:54029)<v229>:1024
2022-12-20 02:09:17.148 [Pooled High Priority Message Processor 15] INFO org.apache.geode.distributed.internal.membership.gms.Services - View with removed and left members removed is View[10.4.20.34(20.34-L1:12493:locator)<ec><v0>:1024|235] members: [10.4.20.34(20.34-L1:12493:locator)<ec><v0>:1024, 10.4.20.35(20.35-L1:13244:locator)<ec><v1>:1024, 10.4.20.148(20.148-S1:54029)<v229>:1024{lead}, 10.4.20.35(20.35-S1:67284)<v230>:1025, 10.4.20.146(20.146-S1:125081)<v232>:1024, 10.4.20.147(20.147-S1:22219)<v234>:1024] and coordinator would be 10.4.20.34(20.34-L1:12493:locator)<ec><v0>:1024
2022-12-20 02:09:17.251 [unicast receiver,95306-YY-V020148-64048] INFO org.apache.geode.distributed.internal.membership.gms.Services - received leave request from 10.4.20.34(20.34-S1:121257)<v203>:1025 for 10.4.20.34(20.34-S1:121257)<v203>:1025
2022-12-20 02:09:17.252 [unicast receiver,95306-YY-V020148-64048] INFO org.apache.geode.distributed.internal.membership.gms.Services - JoinLeave.processMessage(LeaveRequestMessage) invoked. isCoordinator=false; isStopping=false; cancelInProgress=false
2022-12-20 02:09:17.252 [unicast receiver,95306-YY-V020148-64048] INFO org.apache.geode.distributed.internal.membership.gms.Services - Checking to see if I should become coordinator. My address is 10.4.20.148(20.148-S1:54029)<v229>:1024
2022-12-20 02:09:17.252 [unicast receiver,95306-YY-V020148-64048] INFO org.apache.geode.distributed.internal.membership.gms.Services - View with removed and left members removed is View[10.4.20.34(20.34-L1:12493:locator)<ec><v0>:1024|235] members: [10.4.20.34(20.34-L1:12493:locator)<ec><v0>:1024, 10.4.20.35(20.35-L1:13244:locator)<ec><v1>:1024, 10.4.20.148(20.148-S1:54029)<v229>:1024{lead}, 10.4.20.35(20.35-S1:67284)<v230>:1025, 10.4.20.146(20.146-S1:125081)<v232>:1024, 10.4.20.147(20.147-S1:22219)<v234>:1024] and coordinator would be 10.4.20.34(20.34-L1:12493:locator)<ec><v0>:1024
2022-12-20 02:09:17.451 [unicast receiver,95306-YY-V020148-64048] INFO org.apache.geode.distributed.internal.membership.gms.Services - received new view: View[10.4.20.34(20.34-L1:12493:locator)<ec><v0>:1024|235] members: [10.4.20.34(20.34-L1:12493:locator)<ec><v0>:1024, 10.4.20.35(20.35-L1:13244:locator)<ec><v1>:1024, 10.4.20.148(20.148-S1:54029)<v229>:1024{lead}, 10.4.20.35(20.35-S1:67284)<v230>:1025, 10.4.20.146(20.146-S1:125081)<v232>:1024, 10.4.20.147(20.147-S1:22219)<v234>:1024] shutdown: [10.4.20.34(20.34-S1:121257)<v203>:1025]
old view is: View[10.4.20.34(20.34-L1:12493:locator)<ec><v0>:1024|234] members: [10.4.20.34(20.34-L1:12493:locator)<ec><v0>:1024, 10.4.20.35(20.35-L1:13244:locator)<ec><v1>:1024, 10.4.20.34(20.34-S1:121257)<v203>:1025{lead}, 10.4.20.148(20.148-S1:54029)<v229>:1024, 10.4.20.35(20.35-S1:67284)<v230>:1025, 10.4.20.146(20.146-S1:125081)<v232>:1024, 10.4.20.147(20.147-S1:22219)<v234>:1024]
2022-12-20 02:09:17.453 [Geode View Processor1] INFO org.apache.geode.distributed.internal.ClusterDistributionManager - Member at 10.4.20.34(20.34-S1:121257)<v203>:1025 gracefully left the distributed cache: departed membership view
2022-12-20 02:09:17.453 [Geode View Processor1] INFO org.apache.geode.distributed.internal.ClusterOperationExecutors - Marking the SerialQueuedExecutor with id : 3 used by the member 10.4.20.34(20.34-S1:121257)<v203>:1025 to be unused.
2022-12-20 02:09:23.540 [Function Execution Processor12] DEBUG com.wntime.geode.function.OperateDataFunction - start exec logic : OperateDataFunction
2022-12-20 02:09:27.880 [Function Execution Processor11] DEBUG com.wntime.geode.function.OperateDataFunction - start exec logic : OperateDataFunction
2022-12-20 02:09:31.419 [Thread-7] WARN org.apache.geode.distributed.internal.ReplyProcessor21 - 15 seconds have elapsed while waiting for replies: <DistributedCacheOperation$CacheOperationReplyProcessor 11238 waiting for 3 replies from [10.4.20.35(20.35-S1:67284)<v230>:1025, 10.4.20.146(20.146-S1:125081)<v232>:1024, 10.4.20.147(20.147-S1:22219)<v234>:1024]> on 10.4.20.148(20.148-S1:54029)<v229>:1024 whose current membership list is: [[10.4.20.148(20.148-S1:54029)<v229>:1024, 10.4.20.35(20.35-S1:67284)<v230>:1025, 10.4.20.34(20.34-L1:12493:locator)<ec><v0>:1024, 10.4.20.146(20.146-S1:125081)<v232>:1024, 10.4.20.35(20.35-L1:13244:locator)<ec><v1>:1024, 10.4.20.147(20.147-S1:22219)<v234>:1024]]
2022-12-20 02:09:31.419 [Thread-8] WARN org.apache.geode.distributed.internal.ReplyProcessor21 - 15 seconds have elapsed while waiting for replies: <DistributedCacheOperation$CacheOperationReplyProcessor 11239 waiting for 3 replies from [10.4.20.35(20.35-S1:67284)<v230>:1025, 10.4.20.146(20.146-S1:125081)<v232>:1024, 10.4.20.147(20.147-S1:22219)<v234>:1024]> on 10.4.20.148(20.148-S1:54029)<v229>:1024 whose current membership list is: [[10.4.20.148(20.148-S1:54029)<v229>:1024, 10.4.20.35(20.35-S1:67284)<v230>:1025, 10.4.20.34(20.34-L1:12493:locator)<ec><v0>:1024, 10.4.20.146(20.146-S1:125081)<v232>:1024, 10.4.20.35(20.35-L1:13244:locator)<ec><v1>:1024, 10.4.20.147(20.147-S1:22219)<v234>:1024]]
2022-12-20 02:09:31.421 [Thread-10] WARN org.apache.geode.distributed.internal.ReplyProcessor21 - 15 seconds have elapsed while waiting for replies: <DistributedCacheOperation$CacheOperationReplyProcessor 11240 waiting for 3 replies from [10.4.20.35(20.35-S1:67284)<v230>:1025, 10.4.20.146(20.146-S1:125081)<v232>:1024, 10.4.20.147(20.147-S1:22219)<v234>:1024]> on 10.4.20.148(20.148-S1:54029)<v229>:1024 whose current membership list is: [[10.4.20.148(20.148-S1:54029)<v229>:1024, 10.4.20.35(20.35-S1:67284)<v230>:1025, 10.4.20.34(20.34-L1:12493:locator)<ec><v0>:1024, 10.4.20.146(20.146-S1:125081)<v232>:1024, 10.4.20.35(20.35-L1:13244:locator)<ec><v1>:1024, 10.4.20.147(20.147-S1:22219)<v234>:1024]]
2022-12-20 02:09:31.424 [Thread-9] WARN org.apache.geode.distributed.internal.ReplyProcessor21 - 15 seconds have elapsed while waiting for replies: <DistributedCacheOperation$CacheOperationReplyProcessor 11241 waiting for 3 replies from [10.4.20.35(20.35-S1:67284)<v230>:1025, 10.4.20.146(20.146-S1:125081)<v232>:1024, 10.4.20.147(20.147-S1:22219)<v234>:1024]> on 10.4.20.148(20.148-S1:54029)<v229>:1024 whose current membership list is: [[10.4.20.148(20.148-S1:54029)<v229>:1024, 10.4.20.35(20.35-S1:67284)<v230>:1025, 10.4.20.34(20.34-L1:12493:locator)<ec><v0>:1024, 10.4.20.146(20.146-S1:125081)<v232>:1024, 10.4.20.35(20.35-L1:13244:locator)<ec><v1>:1024, 10.4.20.147(20.147-S1:22219)<v234>:1024]]
2022-12-20 02:09:31.427 [Thread-6] WARN org.apache.geode.distributed.internal.ReplyProcessor21 - 15 seconds have elapsed while waiting for replies: <DistributedCacheOperation$CacheOperationReplyProcessor 11242 waiting for 3 replies from [10.4.20.35(20.35-S1:67284)<v230>:1025, 10.4.20.146(20.146-S1:125081)<v232>:1024, 10.4.20.147(20.147-S1:22219)<v234>:1024]> on 10.4.20.148(20.148-S1:54029)<v229>:1024 whose current membership list is: [[10.4.20.148(20.148-S1:54029)<v229>:1024, 10.4.20.35(20.35-S1:67284)<v230>:1025, 10.4.20.34(20.34-L1:12493:locator)<ec><v0>:1024, 10.4.20.146(20.146-S1:125081)<v232>:1024, 10.4.20.35(20.35-L1:13244:locator)<ec><v1>:1024, 10.4.20.147(20.147-S1:22219)<v234>:1024]]
2
2022-12-20 02:09:42.518 [ClientHealthMonitor Thread] WARN org.apache.geode.internal.cache.tier.sockets.ClientHealthMonitor - Server connection from [identity(10.255.170.107(15:loner):49844:f703d5d6,connection=2; port=45420] is being terminated because its client timeout of 10000 has expired.
2022-12-20 02:09:42.518 [ClientHealthMonitor Thread] WARN org.apache.geode.internal.cache.tier.sockets.ClientHealthMonitor - Server connection from [identity(10.255.28.152(15:loner):45140:3406d5d6,connection=2; port=41590] is being terminated because its client timeout of 10000 has expired.
2022-12-20 02:09:42.519 [ClientHealthMonitor Thread] WARN org.apache.geode.internal.cache.tier.sockets.ClientHealthMonitor - ClientHealthMonitor: Unregistering client with member id identity(10.255.28.152(15:loner):45140:3406d5d6,connection=2 due to: Unknown reason
2022-12-20 02:09:42.519 [ClientHealthMonitor Thread] WARN org.apache.geode.internal.cache.tier.sockets.ClientHealthMonitor - Server connection from [identity(10.255.242.77(13:loner):48228:1706d5d6,connection=2; port=57978] is being terminated because its client timeout of 10000 has expired.
2022-12-20 02:09:42.519 [ClientHealthMonitor Thread] WARN org.apache.geode.internal.cache.tier.sockets.ClientHealthMonitor - ClientHealthMonitor: Unregistering client with member id identity(10.255.242.77(13:loner):48228:1706d5d6,connection=2 due to: Unknown reason
2022-12-20 02:09:42.703 [ServerConnection on port 42973 Thread 61] WARN org.apache.geode.distributed.internal.ReplyProcessor21 - 15 seconds have elapsed while waiting for replies: <DistributedCacheOperation$CacheOperationReplyProcessor 11246 waiting for 3 replies from [10.4.20.35(20.35-S1:67284)<v230>:1025, 10.4.20.146(20.146-S1:125081)<v232>:1024, 10.4.20.147(20.147-S1:22219)<v234>:1024]> on 10.4.20.148(20.148-S1:54029)<v229>:1024 whose current membership list is: [[10.4.20.148(20.148-S1:54029)<v229>:1024, 10.4.20.35(20.35-S1:67284)<v230>:1025, 10.4.20.34(20.34-L1:12493:locator)<ec><v0>:1024, 10.4.20.146(20.146-S1:125081)<v232>:1024, 10.4.20.35(20.35-L1:13244:locator)<ec><v1>:1024, 10.4.20.147(20.147-S1:22219)<v234>:1024]]
2022-12-20 02:09:47.522 [ClientHealthMonitor Thread] WARN org.apache.geode.internal.cache.tier.sockets.ClientHealthMonitor - Server connection from [identity(10.255.16.178(20:loner):35578:8902d5d6,connection=2; port=60302] is being terminated because its client timeout of 10000 has expired.
2022-12-20 02:09:48.527 [ClientHealthMonitor Thread] WARN org.apache.geode.internal.cache.tier.sockets.ClientHealthMonitor - Server connection from [identity(10.255.150.234(39:loner):50436:3406d5d6,connection=2; port=53498] is being terminated because its client timeout of 10000 has expired.
2022-12-20 02:09:51.529 [ClientHealthMonitor Thread] WARN org.apache.geode.internal.cache.tier.sockets.ClientHealthMonitor - Server connection from [identity(10.255.95.181(14:loner):42602:2f02d5d6,connection=2; port=53556] is being terminated because its client timeout of 10000 has expired.
2022-12-20 02:10:25.576 [ClientHealthMonitor Thread] WARN org.apache.geode.internal.cache.tier.sockets.ClientHealthMonitor - Server connection from [identity(10.255.150.234(39:loner):50436:3406d5d6,connection=2; port=56390] is being terminated because its client timeout of 10000 has expired.
2022-12-20 02:10:26.577 [ClientHealthMonitor Thread] WARN org.apache.geode.internal.cache.tier.sockets.ClientHealthMonitor - Server connection from [identity(10.255.16.178(20:loner):35578:8902d5d6,connection=2; port=60832] is being terminated because its client timeout of 10000 has expired.
2022-12-20 02:10:27.392 [Function Execution Processor7] DEBUG com.wntime.geode.function.OperateDataFunction - start exec logic : OperateDataFunction
2022-12-20 02:10:29.380 [ThreadsMonitor] WARN org.apache.geode.internal.monitoring.ThreadsMonitoringProcess - Thread 641 (0x281) is stuck
2022-12-20 02:10:29.396 [ThreadsMonitor] WARN org.apache.geode.internal.monitoring.executor.AbstractExecutor - Thread <641> (0x281) that was executed at <20 Dec 2022 02:09:19 CST> has been stuck for <70.145 seconds> and number of thread monitor iteration <1>
Thread Name <Pooled Serial Message Processor1-1> state <WAITING>
Waiting on <java.util.concurrent.locks.ReentrantReadWriteLock$FairSync@1ce290a7>
Owned By <ServerConnection on port 42973 Thread 63> with ID <930>
Executor Group <SerialQueuedExecutorWithDMStats>
Monitored metric <ResourceManagerStats.numThreadsStuck>
Thread stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)
org.apache.geode.internal.cache.wan.serial.SerialGatewaySenderQueue.put(SerialGatewaySenderQueue.java:223)
org.apache.geode.internal.cache.wan.serial.SerialGatewaySenderEventProcessor.queuePrimaryEvent(SerialGatewaySenderEventProcessor.java:477)
org.apache.geode.internal.cache.wan.serial.SerialGatewaySenderEventProcessor.enqueueEvent(SerialGatewaySenderEventProcessor.java:445)
org.apache.geode.internal.cache.wan.serial.ConcurrentSerialGatewaySenderEventProcessor.enqueueEvent(ConcurrentSerialGatewaySenderEventProcessor.java:162)
org.apache.geode.internal.cache.wan.serial.ConcurrentSerialGatewaySenderEventProcessor.enqueueEvent(ConcurrentSerialGatewaySenderEventProcessor.java:116)
org.apache.geode.internal.cache.wan.AbstractGatewaySender.distribute(AbstractGatewaySender.java:1082)
org.apache.geode.internal.cache.LocalRegion.notifyGatewaySender(LocalRegion.java:6141)
org.apache.geode.internal.cache.LocalRegion.basicPutPart2(LocalRegion.java:5777)
org.apache.geode.internal.cache.map.RegionMapPut.doBeforeCompletionActions(RegionMapPut.java:282)
org.apache.geode.internal.cache.map.AbstractRegionMapPut.doPutAndDeliverEvent(AbstractRegionMapPut.java:301)
org.apache.geode.internal.cache.map.AbstractRegionMapPut$$Lambda$258/445977285.run(Unknown Source)
org.apache.geode.internal.cache.map.AbstractRegionMapPut.runWithIndexUpdatingInProgress(AbstractRegionMapPut.java:308)
org.apache.geode.internal.cache.map.AbstractRegionMapPut.doPutIfPreconditionsSatisified(AbstractRegionMapPut.java:296)
org.apache.geode.internal.cache.map.AbstractRegionMapPut.doPutOnSynchronizedRegionEntry(AbstractRegionMapPut.java:282)
org.apache.geode.internal.cache.map.AbstractRegionMapPut.doPutOnRegionEntryInMap(AbstractRegionMapPut.java:273)
org.apache.geode.internal.cache.map.AbstractRegionMapPut.addRegionEntryToMapAndDoPut(AbstractRegionMapPut.java:251)
org.apache.geode.internal.cache.map.AbstractRegionMapPut.doPutRetryingIfNeeded(AbstractRegionMapPut.java:216)
org.apache.geode.internal.cache.map.AbstractRegionMapPut$$Lambda$256/2071190751.run(Unknown Source)
org.apache.geode.internal.cache.map.AbstractRegionMapPut.doWithIndexInUpdateMode(AbstractRegionMapPut.java:198)
org.apache.geode.internal.cache.map.AbstractRegionMapPut.doPut(AbstractRegionMapPut.java:180)
org.apache.geode.internal.cache.map.AbstractRegionMapPut$$Lambda$255/113991150.run(Unknown Source)
org.apache.geode.internal.cache.map.AbstractRegionMapPut.runWhileLockedForCacheModification(AbstractRegionMapPut.java:119)
org.apache.geode.internal.cache.map.RegionMapPut.runWhileLockedForCacheModification(RegionMapPut.java:161)
org.apache.geode.internal.cache.map.AbstractRegionMapPut.put(AbstractRegionMapPut.java:169)
org.apache.geode.internal.cache.AbstractRegionMap.basicPut(AbstractRegionMap.java:2044)
org.apache.geode.internal.cache.LocalRegion.virtualPut(LocalRegion.java:5602)
org.apache.geode.internal.cache.DistributedRegion.virtualPut(DistributedRegion.java:387)
org.apache.geode.internal.cache.LocalRegionDataView.putEntry(LocalRegionDataView.java:170)
org.apache.geode.internal.cache.LocalRegion.basicUpdate(LocalRegion.java:5573)
org.apache.geode.internal.cache.AbstractUpdateOperation.doPutOrCreate(AbstractUpdateOperation.java:156)
org.apache.geode.internal.cache.AbstractUpdateOperation$AbstractUpdateMessage.basicOperateOnRegion(AbstractUpdateOperation.java:307)
org.apache.geode.internal.cache.DistributedPutAllOperation$PutAllMessage.doEntryPut(DistributedPutAllOperation.java:1114)
org.apache.geode.internal.cache.DistributedPutAllOperation$PutAllMessage$1.run(DistributedPutAllOperation.java:1194)
org.apache.geode.internal.cache.event.DistributedEventTracker.syncBulkOp(DistributedEventTracker.java:481)
Lock owner thread stack
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
org.apache.geode.internal.util.concurrent.StoppableCountDownLatch.await(StoppableCountDownLatch.java:72)
org.apache.geode.distributed.internal.ReplyProcessor21.basicWait(ReplyProcessor21.java:731)
org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:802)
org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:779)
org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:865)
org.apache.geode.internal.cache.DistributedCacheOperation.waitForAckIfNeeded(DistributedCacheOperation.java:779)
org.apache.geode.internal.cache.DistributedCacheOperation._distribute(DistributedCacheOperation.java:676)
org.apache.geode.internal.cache.DistributedCacheOperation.startOperation(DistributedCacheOperation.java:277)
org.apache.geode.internal.cache.DistributedCacheOperation.distribute(DistributedCacheOperation.java:318)
org.apache.geode.internal.cache.DistributedRegion.distributeUpdate(DistributedRegion.java:514)
org.apache.geode.internal.cache.DistributedRegion.basicPutPart3(DistributedRegion.java:492)
org.apache.geode.internal.cache.map.RegionMapPut.doAfterCompletionActions(RegionMapPut.java:307)
org.apache.geode.internal.cache.map.AbstractRegionMapPut.doPut(AbstractRegionMapPut.java:185)
org.apache.geode.internal.cache.map.AbstractRegionMapPut$$Lambda$255/113991150.run(Unknown Source)
org.apache.geode.internal.cache.map.AbstractRegionMapPut.runWhileLockedForCacheModification(AbstractRegionMapPut.java:119)
org.apache.geode.internal.cache.map.RegionMapPut.runWhileLockedForCacheModification(RegionMapPut.java:161)
org.apache.geode.internal.cache.map.AbstractRegionMapPut.put(AbstractRegionMapPut.java:169)
org.apache.geode.internal.cache.AbstractRegionMap.basicPut(AbstractRegionMap.java:2044)
org.apache.geode.internal.cache.LocalRegion.virtualPut(LocalRegion.java:5602)
org.apache.geode.internal.cache.DistributedRegion.virtualPut(DistributedRegion.java:387)
org.apache.geode.internal.cache.wan.serial.SerialGatewaySenderQueue$SerialGatewaySenderQueueMetaRegion.virtualPut(SerialGatewaySenderQueue.java:1215)
org.apache.geode.internal.cache.LocalRegion.virtualPut(LocalRegion.java:5580)
org.apache.geode.internal.cache.LocalRegionDataView.putEntry(LocalRegionDataView.java:156)
org.apache.geode.internal.cache.LocalRegion.basicPut(LocalRegion.java:5038)
org.apache.geode.internal.cache.LocalRegion.validatedPut(LocalRegion.java:1637)
org.apache.geode.internal.cache.LocalRegion.put(LocalRegion.java:1624)
org.apache.geode.internal.cache.AbstractRegion.put(AbstractRegion.java:442)
org.apache.geode.internal.cache.wan.serial.SerialGatewaySenderQueue.putAndGetKey(SerialGatewaySenderQueue.java:245)
org.apache.geode.internal.cache.wan.serial.SerialGatewaySenderQueue.put(SerialGatewaySenderQueue.java:232)
org.apache.geode.internal.cache.wan.serial.SerialGatewaySenderEventProcessor.queuePrimaryEvent(SerialGatewaySenderEventProcessor.java:477)
org.apache.geode.internal.cache.wan.serial.SerialGatewaySenderEventProcessor.enqueueEvent(SerialGatewaySenderEventProcessor.java:445)
org.apache.geode.internal.cache.wan.serial.ConcurrentSerialGatewaySenderEventProcessor.enqueueEvent(ConcurrentSerialGatewaySenderEventProcessor.java:162)
org.apache.geode.internal.cache.wan.serial.ConcurrentSerialGatewaySenderEventProcessor.enqueueEvent(ConcurrentSerialGatewaySenderEventProcessor.java:116)
org.apache.geode.internal.cache.wan.AbstractGatewaySender.distribute(AbstractGatewaySender.java:1082)
org.apache.geode.internal.cache.LocalRegion.notifyGatewaySender(LocalRegion.java:6141)
2022-12-20 02:10:29.397 [ThreadsMonitor] WARN org.apache.geode.internal.monitoring.ThreadsMonitoringProcess - Thread 646 (0x286) is stuck
2022-12-20 02:10:29.413 [ThreadsMonitor] WARN org.apache.geode.internal.monitoring.executor.AbstractExecutor - Thread <646> (0x286) that was executed at <20 Dec 2022 02:09:16 CST> has been stuck for <72.891 seconds> and number of thread monitor iteration <1>
Thread Name <Pooled Serial Message Processor2-1> state <WAITING>
Waiting on <java.util.concurrent.locks.ReentrantReadWriteLock$FairSync@1ce290a7>
Owned By <ServerConnection on port 42973 Thread 63> with ID <930>
Executor Group <SerialQueuedExecutorWithDMStats>
Monitored metric <ResourceManagerStats.numThreadsStuck>
Thread stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)
org.apache.geode.internal.cache.wan.serial.SerialGatewaySenderQueue.put(SerialGatewaySenderQueue.java:223)
org.apache.geode.internal.cache.wan.serial.SerialGatewaySenderEventProcessor.queuePrimaryEvent(SerialGatewaySenderEventProcessor.java:477)
org.apache.geode.internal.cache.wan.serial.SerialGatewaySenderEventProcessor.enqueueEvent(SerialGatewaySenderEventProcessor.java:445)
org.apache.geode.internal.cache.wan.serial.ConcurrentSerialGatewaySenderEventProcessor.enqueueEvent(ConcurrentSerialGatewaySenderEventProcessor.java:162)
org.apache.geode.internal.cache.wan.serial.ConcurrentSerialGatewaySenderEventProcessor.enqueueEvent(ConcurrentSerialGatewaySenderEventProcessor.java:116)
org.apache.geode.internal.cache.wan.AbstractGatewaySender.distribute(AbstractGatewaySender.java:1082)
org.apache.geode.internal.cache.LocalRegion.notifyGatewaySender(LocalRegion.java:6141)
org.apache.geode.internal.cache.LocalRegion.basicPutPart2(LocalRegion.java:5777)
org.apache.geode.internal.cache.map.RegionMapPut.doBeforeCompletionActions(RegionMapPut.java:282)
org.apache.geode.internal.cache.map.AbstractRegionMapPut.doPutAndDeliverEvent(AbstractRegionMapPut.java:301)
There are 86 stuck threads in this node
The cluster uses WAN gateway, which is a two-way data transmission architecture. When I restart the main sending gateway, the cluster fails, the cluster cannot write data, there are threads stuck, over time, finally 86 threads are stuck, the cluster after 57 minutes, all the stuck threads are restored, during this period has been reporting exceptions in the log, can not be recovered. Hope to know what causes this and find a quick way to recover.
