Acquired Connections grows abruptly leading to server crash

Question

I have a Java application that schedules a cron job after every 1 min. It runs on Glassfish 4. We are using Hibernate with JTA Entity Manager which is container managed for executing the queries on SQL Server database.

JDBC Connection Pool Settings are:

Initial and Minimum Pool Size:16
Maximum Pool Size:64
Pool Resize Quantity:4
Idle Timeout:300
Max Wait Time:60000

JDBC Connection Pool Statistics after 22 Hours run:

NumConnUsed 0count
NumConnAcquired 14404count
NumConnReleased 14404count NumConnCreated 16count
NumConnFree 16count

The number of acquired connections keeps on incrementing and the Glassfish 4 crashes after around 10 days with below exception.

RAR5117 : Failed to obtain/create connection from connection pool [ com.beonic.tiv5 ]. Reason : com.sun.appserv.connectors.internal.api.PoolingException: java.lang.RuntimeException: Got exception during XAResource.start:

Please suggest how to avoid Glassfish crash.

Can you put the code of the job running? Are you closing closing the persistence mananger ? — Gatusko, Dec 08 '16 at 13:04
According to the documentation "The Container Managed Persistence Context - as the name states - is managed by the enterprise container. The container is responsible for Persistence Context injection into enterprise components, and is also responsible for its disposal at the end of the current transaction." We cannot explicitly close entity managers in container managed transactions as it will throw IllegalStateException. — Rashmi, Dec 08 '16 at 13:33
This is an example of the sample code: public Insight findInsightByName(String name) { Context ic; EntityManager em; Insight loc = null; try { ic = new InitialContext(); em = (EntityManager) ic.lookup(kTIv5PU); loc = (Insight) em.createQuery("select d from Insight d WHERE d.name = :name ") .setParameter("name", name).getSingleResult(); } catch (NamingException ex) { Logger.getLogger(TInsightDAO.class.getName()).log(Level.SEVERE, null, ex); } finally { em = null; ic = null; } return loc; } — Rashmi, Dec 08 '16 at 13:36
Do you get any stacktrace in the logs? What is `com.beonic.tiv5` - are you using some custom connection pool? What does it mean that your Glassfish crashes - the JVM completely crashes? — OndroMih, Dec 09 '16 at 08:40
The cron job is scheduled using quartz scheduler. com.beonic.tiv5 is the name of datasource in JDBC connection pooling. Glassfish crashes means the server stops with exception that no more connection is available to process the request. — Rashmi, Dec 10 '16 at 08:00

Gatusko · Answer 1 · 2016-12-08T14:02:53.663

finally 
{ 
em = null;
 ic = null; 
}

I think here is the problem you are never commiting or closing the transacction Giving this example and documentation of JTA check 5.2.2

// BMT idiom
@Resource public UserTransaction utx;
@Resource public EntityManagerFactory factory;

public void doBusiness() {
    EntityManager em = factory.createEntityManager();
    try {

    // do some work
    ...

    utx.commit();
}
catch (RuntimeException e) {
    if (utx != null) utx.rollback();
    throw e; // or display error message
}
finally {
    em.close();
}

This is the correct way of doing a transacction. But you are only nulling the values and nothing more, that's why you your pools and not being closed Here is more documentation about Transactions

The example you have quoted is for bean-managed transactions (BMT). I am using Container Managed Transactions (CMT). The document link which you have sent contains information about this. All closing and rollback is managed by the container. If we try to close explicitly, it will throw IllegalStateException. — Rashmi, Dec 09 '16 at 05:08

score 0 · Answer 2 · answered Dec 09 '16 at 08:48

0

It's hard to tell what is the real cause of the problem, but the problem might be that all your connections have become stale because not used for a long time.

It is a good practice to set up connection validation, which ensures that connections are reopened when closed by the external server.

There is a thorough article about connection pools in Glassfish/Payara, checkout especially the section about Connection validation (using Derby DB in the example):

To turn on connection validation :

asadmin set resources.jdbc-connection-pool.test-pool.connection-validation-method=custom-validation

asadmin set resources.jdbc-connection-pool.test-pool.validation-classname= org.glassfish.api.jdbc.validation.DerbyConnectionValidation

asadmin set resources.jdbc-connection-pool.test-pool.is-connection-validation-required=true

answered Dec 09 '16 at 08:48

OndroMih

7,280
1
26
44

I will turn on the connection validation and test. Do you have idea how the hibernate entity manager in container managed transactions(CMT) deals with opening and closing connections. Does it create connection for every query executed. – Rashmi Dec 10 '16 at 08:07
In container managed transactions, hibernate just asks for a connection from the container every time (I'm not sure whether for each query or once per transaction, but I guess only once per transaction). It does not deal with opening/closing them, it just gets a connection. If it is stale, it cannot do anything just throw an error. The server should make sure the connection is not stale and renew it. It may do it in many ways - from renewing idle connections to validation of the connection by issuing a test SQL before each usage. – OndroMih Dec 12 '16 at 09:42
"The server should make sure the connection is not stale and renew it. It may do it in many ways - from renewing idle connections to validation of the connection by issuing a test SQL before each usage." Can you please elaborate how to renew idle connections in JTA. – Rashmi Dec 13 '16 at 05:32
By default, GlassFish/Payara don't renew any connection. This has to be turned on on the JDBC Connection Pool. Best is to turn on validation either using table or custom, if available for your DB. Also, turning on connection leak detection should help. For more details, please read the blog post linked in the answer. – OndroMih Dec 13 '16 at 15:46

Acquired Connections grows abruptly leading to server crash

2 Answers2