InitialContext.lookup of remote EJB fails after remote server restart

Question

We have a setup in which EJB A is running on server A, and another EJB B is running on server B. EJB A connects to EJB B via IIOP. This setup normally works, but if server B is restarted, EJB A will fail until server A is restarted too.

The issue is that if server B is restarted, all calls to InitialContext.lookup by EJB A fails with a "java.io.IOException: End-of-stream" exception until server A is restarted. I wasn't able to find information on whether our app server (GlassFish) does any kind of caching for InitialContext.lookup. Are there any other reasons why lookups would fail until a server restart? If InitialContext.lookup does cache connections, how would I get around that?

Our servers run Sun Application Server 9.1. The lookup is actually done via org.springframework.jndi.JndiTemplate, but the stack trace says that the JndiTemplate is calling InitialContext.lookup().

Thanks for any insight.

P.S. I should clarify that I'm trying to figure out whether it's possible to avoid having to restart server A every time server B is restarted.

Definition of JndiTemplate (with some text blacked out with 'x's and '#'s)

<bean id="xxxxxxxxxx"  class="org.springframework.jndi.JndiTemplate">
  <property name="environment">
   <props>
    <prop key="java.naming.factory.url.pkgs">com.sun.enterprise.naming</prop>
    <prop key="java.naming.factory.initial">com.sun.enterprise.naming.SerialInitContextFactory</prop>
    <prop key="java.naming.provider.url">iiop://xxxxxxxxxx:####</prop>
    <prop key="java.naming.factory.state">com.sun.corba.ee.impl.presentation.rmi.JNDIStateFactoryImpl</prop>
    <prop key="org.omg.CORBA.ORBInitialHost">xxxxxxxxxx</prop>
    <prop key="org.omg.CORBA.ORBInitialPort">####</prop>
   </props>
  </property>
 </bean>

And the stack trace (with one part replaced with '[application methods]'):

NAM0004: Exception during name lookup : {0}
java.rmi.MarshalException: CORBA COMM_FAILURE 1398079696 Maybe; nested exception is: 
 org.omg.CORBA.COMM_FAILURE:   vmcid: SUN  minor code: 208 completed: Maybe
 at com.sun.corba.ee.impl.javax.rmi.CORBA.Util.mapSystemException(Util.java:271)
 at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:205)
 at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:152)
 at com.sun.corba.ee.impl.presentation.rmi.bcel.BCELStubBase.invoke(BCELStubBase.java:225)
 at com.sun.enterprise.naming._SerialContextProvider_DynamicStub.lookup(com/sun/enterprise/naming/_SerialContextProvider_DynamicStub.java)
 at com.sun.enterprise.naming.SerialContext.lookup(SerialContext.java:398)
 at javax.naming.InitialContext.lookup(InitialContext.java:351)
 at org.springframework.jndi.JndiTemplate$1.doInContext(JndiTemplate.java:155)
 at org.springframework.jndi.JndiTemplate.execute(JndiTemplate.java:88)
 at org.springframework.jndi.JndiTemplate.lookup(JndiTemplate.java:153)
 at [application methods]
 at org.springframework.web.servlet.mvc.SimpleFormController.processFormSubmission(SimpleFormController.java:267)
 at org.springframework.web.servlet.mvc.AbstractFormController.handleRequestInternal(AbstractFormController.java:265)
 at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)
 at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
 at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:875)
 at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:809)
 at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:571)
 at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:511)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:738)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
 at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:411)
 at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:299)
 at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:271)
 at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:202)
 at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
 at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
 at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
 at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:206)
 at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
 at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
 at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
 at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
 at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:150)
 at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
 at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
 at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
 at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
 at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:272)
 at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:637)
 at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:568)
 at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:813)
 at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:341)
 at com.sun.enterprise.web.connector.grizzly.ssl.SSLReadTask.process(SSLReadTask.java:440)
 at com.sun.enterprise.web.connector.grizzly.ssl.SSLReadTask.doTask(SSLReadTask.java:228)
 at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265)
 at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106)
Caused by: org.omg.CORBA.COMM_FAILURE:   vmcid: SUN  minor code: 208 completed: Maybe
 at com.sun.corba.ee.impl.logging.ORBUtilSystemException.connectionAbort(ORBUtilSystemException.java:2862)
 at com.sun.corba.ee.impl.logging.ORBUtilSystemException.connectionAbort(ORBUtilSystemException.java:2880)
 at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.doOptimizedReadStrategy(SocketOrChannelConnectionImpl.java:1788)
 at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.doWork(SocketOrChannelConnectionImpl.java:1263)
 at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:555)
Caused by: org.omg.CORBA.COMM_FAILURE:   vmcid: SUN  minor code: 211  completed: No
 at com.sun.corba.ee.impl.logging.ORBUtilSystemException.ioexceptionWhenReadingConnection(ORBUtilSystemException.java:2946)
 at com.sun.corba.ee.impl.logging.ORBUtilSystemException.ioexceptionWhenReadingConnection(ORBUtilSystemException.java:2965)
 at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.nonBlockingRead(SocketOrChannelConnectionImpl.java:2000)
 at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.doOptimizedReadStrategy(SocketOrChannelConnectionImpl.java:1713)
 ... 2 more
Caused by: java.io.IOException: End-of-stream
 at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.nonBlockingRead(SocketOrChannelConnectionImpl.java:1989)
 ... 3 more

I've realized that the problem only occurs if server A tried to connect to EJB B *while* server B is being restarted. If a call was attempted while server B is restarted, then server A will forever be unable to connect until server A is restarted. If server A has been calling EJB B before server B is restarted, it does not make any calls to EJB B while server B is being restarted, and then starts making calls again after server B is back up, everything works fine. — jthg, Oct 20 '10 at 20:25
I'm experiencing the same problem (unable to reconnect forever after accessing while stopped) - did you solve this? — Vetle, Jan 11 '11 at 10:09
No, I never really solved this. I didn't spend much more time on this issue though. We "fixed" it by always restarting server A if server B is restarted. — jthg, Jan 12 '11 at 15:14

score 0 · Answer 1 · answered Oct 24 '14 at 07:19

Method syncChanges() is running periodically for synchonization purpose. The main idea is to close context if error occurs(after restarting server for exsample) and reinitialize it. After closing context with method context.close() you should null it's value.

public class SyncOperationsService {
    private InitialContext context;
    private SyncOperationsRemote syncOperationsRemote = null; //RemoteBean

    public String syncChanges() {
        try {
            if (context == null) {
                context = new InitialContext(properties);
            }

            if (context != null) {
                syncOperationsRemote = (SyncOperationsRemote) context.lookup("PathToYourRemoteBean");
            }

            result = syncOperationsRemote.syncChanges(dbImportList);  //Try execute remote bean

        } catch (Exception e) {
            log.error(e.getMessage());

            try {
                context.close();
            } catch (Exception e2) {
                log.error(e2.getMessage());
            }
            context = null;
        }
    }
}

score 0 · Accepted Answer · answered Oct 20 '10 at 16:35

0

By the stack trace, I can conclude that Spring Framework itself doesn't cache - it actually invokes InitialContext.lookup(). Still, the stack trace and the error code imply that there's a connection breakage.

WebSphere used to have a similar bug back in the days, which was fixed since. Turned out to be a caching problem done by the application server itself. I suspect SunAS has a similar bug in 9.1...

Isaac

answered Oct 20 '10 at 16:35

Isaac

16,458
5
57
81

I'm definitely suspecting that it's an application server bug. I haven't managed to test possible code workarounds yet. – jthg Oct 20 '10 at 20:28

InitialContext.lookup of remote EJB fails after remote server restart

2 Answers2