Ignite Cache Reconnection Issue (Cache is stopped)

Question

Currently we are using Ignite 2.1 and Spring Data Ignite Repository. I have a server node and a client node with one cache "TestCache". When client is connected to server, and server node is down, it will disconnect. When the server node is started back, client node will reconnect back.

However, when it is connected back, the cache "TestCache" is no longer accessible with the following exception:

java.lang.IllegalStateException: class org.apache.ignite.internal.processors.cache.CacheStoppedException: Failed to perform cache operation (cache is stopped): TestCache
    at org.apache.ignite.internal.processors.cache.GridCacheGateway.enter(GridCacheGateway.java:164)
    at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.onEnter(IgniteCacheProxy.java:2722)
    at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.query(IgniteCacheProxy.java:775)
    at org.apache.ignite.springdata.repository.query.IgniteRepositoryQuery.execute(IgniteRepositoryQuery.java:117)
    at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.doInvoke(RepositoryFactorySupport.java:483)
    at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.invoke(RepositoryFactorySupport.java:461)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.data.projection.DefaultMethodInvokingMethodInterceptor.invoke(DefaultMethodInvokingMethodInterceptor.java:56)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.data.repository.core.support.SurroundingTransactionDetectorMethodInterceptor.invoke(SurroundingTransactionDetectorMethodInterceptor.java:57)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
    at com.sun.proxy.$Proxy73.findByVslMAndInVoyNAndOutVoyNAndTypeC(Unknown Source)
    at com.psa.pc.am.stw.component.AMServiceComponent.getSpaVesselProfileData(AMServiceComponent.java:56)
    at com.psa.pc.am.stw.service.TestService.getSpaVesselProfileData(TestService.java:37)
    at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:105)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:478)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:80)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:799)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1455)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)
Caused by: class org.apache.ignite.internal.processors.cache.CacheStoppedException: Failed to perform cache operation (cache is stopped): TestCache
    ... 70 more

Had tried the following way but it does not seems to connect back to the cache. Any idea for the following issue?

ig.events().localListen(new IgnitePredicate<Event>() {
        @Override public boolean apply(Event event) {
            switch (event.type()) {
                case EventType.EVT_CLIENT_NODE_RECONNECTED:
                    logger.info("Ignite Client reconnected!");
                    Runnable task = () -> {
                        ig.getOrCreateCache(spaCacheName);
                        logger.info("getOrCreateCache after reconnected");
                    };

                    Thread thread = new Thread(task);
                    thread.start();
            }
            return true;
        }
    }, EventType.EVT_CLIENT_NODE_DISCONNECTED, EventType.EVT_CLIENT_NODE_RECONNECTED);

Thanks in advance~

Some info i saw regarding this issue: How to restore cache after ignite server reconnected https://issues.apache.org/jira/browse/IGNITE-5789

Also, still facing the issue mentioned in the following JIRA during server node restarts (https://issues.apache.org/jira/browse/IGNITE-1564). This issue supposedly should had been resolved. — Cheng, Sep 05 '17 at 11:35

score 8 · Answer 1 · answered Sep 05 '17 at 12:35

8

Yes, you correctly found an issue related to this problem: https://issues.apache.org/jira/browse/IGNITE-5789

After restart server node(obviously) doesn't have any information about caches started before it was stopped. In the same time, if client reconnects with the EVT_CLIENT_NODE_RECONNECTED event, it doesn't create cache again, because it thinks that it was reconnected to the same cluster, where this cache was already created.

As a workaround, you can restart client node, it will create cache again after normal restart.

answered Sep 05 '17 at 12:35

Evgenii Zhuravlev

2,987
1
9
15

Can the restart be done within the code upon reconnect or disconnect? We r planning to go production, but this dont look too good to go production if i need to manual restart. – Cheng Sep 05 '17 at 12:37
I think that in production you will have more that one server node, so you will never face this exception. Also, yes, it's possible to restart node from code, check Ignition.stop() and start(). – Evgenii Zhuravlev Sep 05 '17 at 12:44
We are starting small with 2 nodes, so it is possible that both are down together. I can try out the stop and start to see how it goes. Think i will listen on reconnect event, then do Ignition stop then Ignition start. Thanks! – Cheng Sep 05 '17 at 12:46
I am using Spring Data Ignite Repository. It seems like its not so straightforward to replace the ignite Spring Bean to the newly started one. – Cheng Sep 05 '17 at 16:58
@cheng did you found good solution for this problem? I'm using preemptible machines on GKE so cache nodes are restarted and I'm getting cache is stopped error. – Mistic92 May 17 '18 at 22:48
@Mistic92 the issue was fixed for Ignite 2.7 and later versions: https://issues.apache.org/jira/browse/IGNITE-5789 – dmagda Aug 26 '20 at 22:58
1

I am still getting this issue. – Atul Agrawal Sep 01 '21 at 06:45

Ignite Cache Reconnection Issue (Cache is stopped)

1 Answers1

Linked