0

I have a Java webapp running on Jetty (version 9.4.20). It uses a custom library for accessing MySql database, which has always worked well.

At random, sometimes, this exception is thrown:

java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: java/lang/IndyBootstrapDispatcher
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:222)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:844)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
        at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:226)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1591)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:542)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:536)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1581)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1307)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:482)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1549)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1204)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:494)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:374)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:268)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:367)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:782)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:918)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError: java/lang/IndyBootstrapDispatcher
        ... 36 more
java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: java/lang/IndyBootstrapDispatcher
        at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1421)
        at com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnection(DefaultConnectionTester.java:286)
        at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.testPooledConnection(C3P0PooledConnectionPool.java:510)
        at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.refurbishResourceOnCheckin(C3P0PooledConnectionPool.java:406)
        at com.mchange.v2.resourcepool.BasicResourcePool.attemptRefurbishResourceOnCheckin(BasicResourcePool.java:1760)
        at com.mchange.v2.resourcepool.BasicResourcePool.access$200(BasicResourcePool.java:44)
        at com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask.run(BasicResourcePool.java:1378)
        at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)
Caused by: java.lang.NoClassDefFoundError: java/lang/IndyBootstrapDispatcher

which freezes the service.

Strange thing is that everything was working and no changes to this library (dependencies, code, ...) or other updates were made.

JDK used is 1.8.0_201.

Now, I know how to handle this kind of exception, but I have not found any reference to java lang IndyBootstrapDispatcher by crawling sources of the interested libraries.

Any idea? Thank you all.

Ghesio
  • 89
  • 11

2 Answers2

1

Jetty is starting with system temp directory for the Java system property java.io.tmpdir.

This is typically the system /tmp directory.

Jetty unpacks wars and accesses deep resources from your webapp from this temp directory.

What happens on some Linux distributions is that the the system temp directory (/tmp) is periodically cleaned up. This can break a running Jetty instance as the content it needs has now been removed out from underneath it.

The key is to use a temp directory for Jetty that isn't the system temp directory.

You can change the temp directory that Jetty uses in a number of ways.

See: Jetty: How to change startup temp directory

Joakim Erdfelt
  • 46,896
  • 7
  • 86
  • 136
  • Thank you for the answer but this is not our case! The directory is set during the startup of server to don't use the `/tmp` directory. On a side note: if `/tmp` is used the least frequently used class are cleaned by linux? – Ghesio Oct 08 '20 at 13:31
  • A server, like Jetty, uses files for as long as the server is running. There's no concept of used "least frequently" from the point of view of a typical `/tmp` cleanup script. It doesn't care that the binary that's using the temp file is still running or not. It just cares about the last updated, which never changes once the server is started. – Joakim Erdfelt Oct 09 '20 at 09:24
  • Keep in mind that you have multiple temp and working directories on a running servlet server. Just changing `/tmp` is often not enough. You have to think about deployment work directory, Java classlib temp, java.io.tmpdir, ServletContext temp, etc ... – Joakim Erdfelt Oct 09 '20 at 09:25
0

Answering my own question if someone gets here.

This is due to ELK apm connector (version used 1.18.0 RC 1 in auto attach mode); by disabling the agent the errors go away. Further tests will be done on different agent versions.

Relevant GitHub issues here and here.

Ghesio
  • 89
  • 11