Random error SQLNonTransientConnectionException: Connection is closed

Question

From time to time we get the following error in our Wildfly Swarm app server (full stack trace at the end):

java.sql.SQLNonTransientConnectionException: (conn=19796385) Connection is closed

Can someone explain how these kind of errors can occur? Why doesn't my persistence layer just create another connection but instead throw this error? The error is totally random and I don't know how to debug this.

I'm guessing that either:

My Hibernate configuration is wrong (do I have to use connection pooling to prevent these errors?)
MariaDB Java Client does something wrong
We are using DeltaSpike Data for simplifying our repository layer. Maybe there is a problem here? (see configuration below)

Server Details

Wildfly Swarm: 2018.4.1
mariadb-java-client: 2.2.3
deltaspike-data: 1.8.0

Wildfly Swarm DataSource Configuration:

swarm:
  datasources:
    data-sources:
      xyzAppDS:
        driver-name: mariadb
        valid-connection-checker-class-name: org.jboss.jca.adapters.jdbc.extensions.mysql.MySQLValidConnectionChecker
        validate-on-match: true
        background-validation: false
        exception-sorter-class-name: org.jboss.jca.adapters.jdbc.extensions.mysql.MySQLExceptionSorter

Hibernate (persistence.xml)

<?xml version="1.0" encoding="UTF-8"?>
    <persistence
            xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
            version="2.1"
            xmlns="http://xmlns.jcp.org/xml/ns/persistence"
            xsi:schemaLocation="http://xmlns.jcp.org/xml/ns/persistence http://xmlns.jcp.org/xml/ns/persistence/persistence_2_1.xsd">
        <persistence-unit name="xyzAppPU" transaction-type="JTA">
            <jta-data-source>java:jboss/datasources/xyzAppDS</jta-data-source>
            <properties>
                <property name="hibernate.dialect" value="org.hibernate.dialect.MySQLDialect"/>
                <property name="javax.persistence.schema-generation.database.action" value=""/>
            </properties>
        </persistence-unit>
    </persistence>

DeltasSpike Data Configuration (apache-deltaspike.properties):

globalAlternatives.org.apache.deltaspike.jpa.spi.transaction.TransactionStrategy=org.apache.deltaspike.jpa.impl.transaction.ContainerManagedTransactionStrategy

EntityManagerProducer for DeltaSpike Data

public class EntityManagerProducer {

    @PersistenceContext
    @Produces
    private EntityManager entityManager;

}

Full Stack Trace

Caused by: javax.persistence.PersistenceException: org.hibernate.exception.JDBCConnectionException: could not extract ResultSet
at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1692)
at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1602)
at org.hibernate.jpa.internal.QueryImpl.getResultList(QueryImpl.java:492)
at org.apache.deltaspike.data.impl.builder.result.QueryProcessorFactory$ListQueryProcessor.executeQuery(QueryProcessorFactory.java:107)
at org.apache.deltaspike.data.impl.handler.CdiQueryInvocationContext.executeQuery(CdiQueryInvocationContext.java:241)
at org.apache.deltaspike.data.impl.builder.AnnotatedQueryBuilder.execute(AnnotatedQueryBuilder.java:53)
at org.apache.deltaspike.data.impl.builder.QueryBuilder.executeQuery(QueryBuilder.java:59)
at org.apache.deltaspike.data.impl.builder.AnnotatedQueryBuilder$Proxy$_$$_WeldClientProxy.executeQuery(Unknown Source)
at org.apache.deltaspike.data.impl.tx.TransactionalQueryRunner.executeNonTransactional(TransactionalQueryRunner.java:66)
at org.apache.deltaspike.data.impl.tx.TransactionalQueryRunner.executeQuery(TransactionalQueryRunner.java:61)
at org.apache.deltaspike.data.impl.tx.TransactionalQueryRunner$Proxy$_$$_WeldClientProxy.executeQuery(Unknown Source)
at org.apache.deltaspike.data.impl.handler.QueryHandler.process(QueryHandler.java:147)
at org.apache.deltaspike.data.impl.handler.QueryHandler.invoke(QueryHandler.java:129)
at org.apache.deltaspike.data.impl.handler.QueryHandler$Proxy$_$$_WeldClientProxy.invoke(Unknown Source)
at org.apache.deltaspike.proxy.impl.invocation.DelegateManualInvocationHandler.proceedOriginal(DelegateManualInvocationHandler.java:47)
at org.apache.deltaspike.proxy.impl.invocation.AbstractManualInvocationHandler.invoke(AbstractManualInvocationHandler.java:59)
at org.apache.deltaspike.proxy.impl.invocation.DelegateManualInvocationHandler$Proxy$_$$_WeldClientProxy.invoke(Unknown Source)
at org.apache.deltaspike.proxy.impl.invocation.DelegateManualInvocationHandler.staticInvoke(DelegateManualInvocationHandler.java:40)
at com.xyzApp.db.repository.PatientNeedRepository$$DSPartialBeanProxy.findIdsForNoOpenNotification(Unknown Source)
at com.xyzApp.business.escalation.EscalationService.escalatePatientNeeds(EscalationService.java:38)
at sun.reflect.GeneratedMethodAccessor464.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
at org.jboss.as.weld.ejb.DelegatingInterceptorInvocationContext.proceed(DelegatingInterceptorInvocationContext.java:92)
at org.jboss.weld.interceptor.proxy.WeldInvocationContext.interceptorChainCompleted(WeldInvocationContext.java:98)
at org.jboss.weld.interceptor.proxy.WeldInvocationContext.proceed(WeldInvocationContext.java:117)
at com.arjuna.ats.jta.cdi.transactional.TransactionalInterceptorBase.invokeInCallerTx(TransactionalInterceptorBase.java:132)
at com.arjuna.ats.jta.cdi.transactional.TransactionalInterceptorRequired.doIntercept(TransactionalInterceptorRequired.java:55)
at com.arjuna.ats.jta.cdi.transactional.TransactionalInterceptorBase.intercept(TransactionalInterceptorBase.java:79)
at com.arjuna.ats.jta.cdi.transactional.TransactionalInterceptorRequired.intercept(TransactionalInterceptorRequired.java:47)
at sun.reflect.GeneratedMethodAccessor437.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73)
at org.jboss.weld.interceptor.proxy.WeldInvocationContext.invokeNext(WeldInvocationContext.java:83)
at org.jboss.weld.interceptor.proxy.WeldInvocationContext.proceed(WeldInvocationContext.java:115)
at org.jboss.weld.bean.InterceptorImpl.intercept(InterceptorImpl.java:108)
at org.jboss.as.weld.ejb.DelegatingInterceptorInvocationContext.proceed(DelegatingInterceptorInvocationContext.java:82)
at org.jboss.as.weld.interceptors.EjbComponentInterceptorSupport.delegateInterception(EjbComponentInterceptorSupport.java:60)
at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:76)
at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:88)
at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:101)
at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.jpa.interceptor.SBInvocationInterceptor.processInvocation(SBInvocationInterceptor.java:47)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3.component.interceptors.NonPooledEJBComponentInstanceAssociatingInterceptor.processInvocation(NonPooledEJBComponentInstanceAssociatingInterceptor.java:59)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:255)
... 111 more
Caused by: org.hibernate.exception.JDBCConnectionException: could not extract ResultSet
at org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:48)
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:111)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:97)
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:79)
at org.hibernate.loader.Loader.getResultSet(Loader.java:2122)
at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1905)
at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1881)
at org.hibernate.loader.Loader.doQuery(Loader.java:925)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:342)
at org.hibernate.loader.Loader.doList(Loader.java:2622)
at org.hibernate.loader.Loader.doList(Loader.java:2605)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2434)
at org.hibernate.loader.Loader.list(Loader.java:2429)
at org.hibernate.loader.custom.CustomLoader.list(CustomLoader.java:336)
at org.hibernate.internal.SessionImpl.listCustomQuery(SessionImpl.java:1980)
at org.hibernate.internal.AbstractSessionImpl.list(AbstractSessionImpl.java:322)
at org.hibernate.internal.SQLQueryImpl.list(SQLQueryImpl.java:125)
at org.hibernate.jpa.internal.QueryImpl.list(QueryImpl.java:606)
at org.hibernate.jpa.internal.QueryImpl.getResultList(QueryImpl.java:483)
... 169 more
Caused by: java.sql.SQLNonTransientConnectionException: (conn=19796385) Connection is closed
at org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.get(ExceptionMapper.java:175)
at org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.getException(ExceptionMapper.java:110)
at org.mariadb.jdbc.MariaDbStatement.executeExceptionEpilogue(MariaDbStatement.java:228)
at org.mariadb.jdbc.MariaDbPreparedStatementClient.executeInternal(MariaDbPreparedStatementClient.java:216)
at org.mariadb.jdbc.MariaDbPreparedStatementClient.execute(MariaDbPreparedStatementClient.java:150)
at org.mariadb.jdbc.MariaDbPreparedStatementClient.executeQuery(MariaDbPreparedStatementClient.java:164)
at org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.executeQuery(WrappedPreparedStatement.java:504)
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:70)
... 184 more
Caused by: java.sql.SQLException: Connection is closed
at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.cmdPrologue(AbstractQueryProtocol.java:1706)
at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:236)
at org.mariadb.jdbc.MariaDbPreparedStatementClient.executeInternal(MariaDbPreparedStatementClient.java:209)
... 188 more

Exactly, if you are not using any connection pooling then you have to check `wait_timeout`, `connection_timeout` of DB. — Amogh, Jun 07 '18 at 07:20
@naren We actually have an EJB Scheduler running that queries the database every 5 minutes.... so there will never be any idle time. — Michael Hunziker, Jun 07 '18 at 11:18
@Amogh Do you still think `wait_timeout` and `connection_timeout` could be relevant even though we do a scheduled query every 5 minutes? How could there be any idle time? — Michael Hunziker, Jun 07 '18 at 11:21
@MichaelHunziker If in case a query is running for long time then yes its relevant? is it a multi-threaded? — Amogh, Jun 07 '18 at 11:45
@MichaelHunziker is there a possibility that scheduler is running on a different node (usually we fix the node to run scheduled tasks). Also you are using MySQLDialect. Can you change it to MariaDBDialect based on your version of db as those are available now? — naren, Jun 07 '18 at 12:10
One more question: does every query fails when you encounter the problem or just some selected queries? And how you get rid of the problem, restart? — naren, Jun 07 '18 at 12:12
Hibernate supports connection url. Connection url can have "sessionVariables=wait_timeout=1000000" in the URL query string — Hackerman, Jun 07 '18 at 18:50
Thanks guys! I'll try your suggestions with the connection pool and the MariaDBDialect — Michael Hunziker, Jun 14 '18 at 05:12
@Amogh: Again I'm not sure if I really have to use third party connection pooling. According to this discussion every JavaEE server should provide connection pooling. We are using Wildfly Swarm 2018.5 (= Wildfly 11): https://developer.jboss.org/thread/241174 — Michael Hunziker, Jun 14 '18 at 13:48

Random error SQLNonTransientConnectionException: Connection is closed

0 Answers0