We've been getting errors occasionally from our test and production environments where ServiceLocatorImpl has shut down which causes it to fail to create stateless EJBs from any @Schedule methods.
javax.ejb.EJBException: javax.ejb.EJBException: javax.ejb.CreateException: Could not create stateless EJB
at com.sun.ejb.containers.StatelessSessionContainer._getContext(StatelessSessionContainer.java:399)
at com.sun.ejb.containers.BaseContainer.getContext(BaseContainer.java:2607)
at com.sun.ejb.containers.BaseContainer.preInvoke(BaseContainer.java:2024)
at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:210)
at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:90)
...
Caused by: java.lang.IllegalStateException: ServiceLocatorImpl(__HK2_Generated_9,12,566324949) has been shut down
at org.jvnet.hk2.internal.ServiceLocatorImpl.checkState(ServiceLocatorImpl.java:2384)
at org.jvnet.hk2.internal.ServiceLocatorImpl.inject(ServiceLocatorImpl.java:997)
at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.inject(AbstractHk2InjectionManager.java:207)
at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.inject(ImmediateHk2InjectionManager.java:30)
at org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider$InjectionManagerInjectedCdiTarget.inject(CdiComponentProvider.java:668)
at org.glassfish.weld.services.JCDIServiceImpl$JCDIInjectionContextImpl.inject(JCDIServiceImpl.java:581)
at org.glassfish.weld.services.JCDIServiceImpl.injectEJBInstance(JCDIServiceImpl.java:328)
at com.sun.ejb.containers.BaseContainer.injectEjbInstance(BaseContainer.java:1818)
at com.sun.ejb.containers.StatelessSessionContainer.createStatelessEJB(StatelessSessionContainer.java:437)
... 88 more
Most of the application continues to work fine and and users can continue using it but background processes that are run from @Schedule all fail until we restart Payara.
We originally ran into this in our test environment after it had been running for multiple days under a fairly high load while we were on Payara 5.2021.4 (uses Jersey 2.30). We found servicelocator impl shutdown issue in jersey 2.22 talking about this issue and it seemed to indicate that Jersey 2.26 fixed the issue.
Since we were already on a later version we dug into the release notes and found that in hk2 2.6.1 there was a fix "Fix for child service locator shutdown" (https://github.com/eclipse-ee4j/glassfish-hk2/blob/master/CHANGELOG) and that Jersey 2.29.1 had updated the hk2 version from 5.1.0 to 6.2.1 (https://github.com/eclipse-ee4j/jersey/pull/4234) and that we had been listing Jersey 2.29 as a dependency. So we updated our versions to Payara 5.2021.8, Jersey 2.34 (hk2 2.6.1) and made our environment restart payara every night. After that we didn't seen the error for a while but since then we've still gotten it a couple times in production.
We haven't been able to reproduce the issue on demand or pin down any other errors that could be causing it. Has anyone else run into this issue on the current Payara and come up with a way to fix it, work around it or mitigate it short of restarting payara whenever it happens?
