After some time of restarting the ignite server and client nodes, client node is disconnecting from server node. We are getting the below logs on both client and servers nodes.
Can any one please guide us what is the rootcause of this issue and how to avoid this issue.
Ignite client logs:
22 Feb 2022 02:43:08,252 [tcp-client-disco-sock-writer-#2%CasinoApacheIgniteServices%] DEBUG tcp.TcpDiscoverySpi - Failed to get acknowledge for message, willtry to reconnect [msg=TcpDiscoveryClientHeartbeatMessage [super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=50b46df1f71-686bbebc-0e60-46d7-a56e-ca617c108462, verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null, isClient=true]], timeout=10000] 22 Feb 2022 02:44:49,204 [tcp-client-disco-sock-writer-#2%CasinoApacheIgniteServices%] ERROR tcp.TcpDiscoverySpi - Failed to send message: null java.io.IOException: Failed to get acknowledge for message: TcpDiscoveryClientHeartbeatMessage [super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=50b46df1f 71-686bbebc-0e60-46d7-a56e-ca617c108462, verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null, isClient=true]] at org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketWriter.body(ClientImpl.java:1118) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
22 Feb 2022 02:44:52,930 [tcp-client-disco-sock-reader-#3%CasinoApacheIgniteServices%] DEBUG tcp.TcpDiscoverySpi - Message has been received: TcpDiscoveryNodeLeftMessage [super=TcpDiscoveryAbstractMessage [sndNodeId=0d80cb6c-2b6a-4173-8c1b-a2ca4d9d9e40, id=69766df1f71-686bbebc-0e60-46d7-a56e-ca617c108462, verifierNodeId=ee48c6dd-c83c-4a84-bff6-8ad53ead251b, topVer=5, pendingIdx=0, failedNodes=null, isClient=true]] 22 Feb 2022 02:44:52,930 [tcp-client-disco-msg-worker-#4%CasinoApacheIgniteServices%] DEBUG tcp.TcpDiscoverySpi - Received node left message for local node: TcpDiscoveryNodeLeftMessage [super=TcpDiscoveryAbstractMessage [sndNodeId=0d80cb6c-2b6a-4173-8c1b-a2ca4d9d9e40, id=69766df1f71-686bbebc-0e60-46d7-a56e-ca617c108462, verifierNodeId=ee48c6dd-c83c-4a84-bff6-8ad53ead251b, topVer=5, pendingIdx=0, failedNodes=null, isClient=true]] 22 Feb 2022 02:44:52,930 [tcp-client-disco-sock-writer-#2%CasinoApacheIgniteServices%] DEBUG tcp.TcpDiscoverySpi - Caught interrupted exception: java.lang.InterruptedException 22 Feb 2022 02:44:52,930 [tcp-client-disco-sock-writer-#2%CasinoApacheIgniteServices%] DEBUG tcp.TcpDiscoverySpi - Grid runnable finished due to interruption without cancellation: tcp-client-disco-sock-writer-#2%CasinoApacheIgniteServices%
22 Feb 2022 02:44:52,960 [framework-worker-group-thread-98] ERROR DFS_LOGGER - Exception in RequestManager genericDataFeed retrieving JSON java.lang.IllegalStateException: Cache has been stopped: filteredFeedObjCacheIgniteNew at org.apache.ignite.internal.processors.cache.GridCacheGateway.checkState(GridCacheGateway.java:85) at org.apache.ignite.internal.processors.cache.GridCacheGateway.enter(GridCacheGateway.java:173) at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.onEnter(IgniteCacheProxy.java:2265) at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.put(IgniteCacheProxy.java:1354) at
Ignite server logs:
[02:44:52,922][INFO ][disco-event-worker-#16%CasinoApacheIgniteServices%][GridDiscoveryManager] Node left topology: TcpDiscoveryNode [id=686bbebc-0e60-46d7-a56e-ca617c108462, addrs=[10.1.208.165, 127.0.0.1], sockAddrs=[hy1ic1rclf001.ivycomptech.co.in/10.1.208.165:0, /127.0.0.1:0], discPort=0, order=4, intOrder=4, lastExchangeTime=1645506498259, loc=false, ver=1.8.0#20161205-sha1:9ca40dbe, isClient=true]
Thanks in advance.
