While upgrading Openshift from 3.11 to 4.10 we are facing intermediate timeouts during direct DB connection initialization, which causing health checks failures and pods restarts. The application running on spring boot 2.6.1, apache commons-dbcp 1.4 is used for managing the connection pool. Same application with exactly same configuration is running with no issues on Openshift version 3.11.
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (IO Error: The Network Adapter could not establish the connection)
at org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1549) ~[commons-dbcp-1.4.jar!/:1.4]
at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1388) ~[commons-dbcp-1.4.jar!/:1.4]
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044) ~[commons-dbcp-1.4.jar!/:1.4]
at org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection(DataSourceUtils.java:159) ~[spring-jdbc-5.3.15.jar!/:5.3.15]
at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:117) ~[spring-jdbc-5.3.15.jar!/:5.3.15]
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) ~[spring-jdbc-5.3.15.jar!/:5.3.15]
... 128 common frames omitted
Caused by: java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:458) ~[ojdbc6-11.2.0.3.jar!/:11.2.0.3.0]
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:546) ~[ojdbc6-11.2.0.3.jar!/:11.2.0.3.0]
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:236) ~[ojdbc6-11.2.0.3.jar!/:11.2.0.3.0]
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32) ~[ojdbc6-11.2.0.3.jar!/:11.2.0.3.0]
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:521) ~[ojdbc6-11.2.0.3.jar!/:11.2.0.3.0]
at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38) ~[commons-dbcp-1.4.jar!/:1.4]
at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582) ~[commons-dbcp-1.4.jar!/:1.4]
at org.apache.commons.dbcp.BasicDataSource.validateConnectionFactory(BasicDataSource.java:1556) ~[commons-dbcp-1.4.jar!/:1.4]
at org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1545) ~[commons-dbcp-1.4.jar!/:1.4]
... 133 common frames omitted
Caused by: oracle.net.ns.NetException: The Network Adapter could not establish the connection
at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:392) ~[ojdbc6-11.2.0.3.jar!/:11.2.0.3.0]
at oracle.net.resolver.AddrResolution.resolveAndExecute(AddrResolution.java:434) ~[ojdbc6-11.2.0.3.jar!/:11.2.0.3.0]
at oracle.net.ns.NSProtocol.establishConnection(NSProtocol.java:687) ~[ojdbc6-11.2.0.3.jar!/:11.2.0.3.0]
at oracle.net.ns.NSProtocol.connect(NSProtocol.java:343) ~[ojdbc6-11.2.0.3.jar!/:11.2.0.3.0]
at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1102) ~[ojdbc6-11.2.0.3.jar!/:11.2.0.3.0]
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:320) ~[ojdbc6-11.2.0.3.jar!/:11.2.0.3.0]
... 141 common frames omitted
Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_372]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_372]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_372]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_372]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_372]
at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_372]
at oracle.net.nt.TcpNTAdapter.connect(TcpNTAdapter.java:150) ~[ojdbc6-11.2.0.3.jar!/:11.2.0.3.0]
at oracle.net.nt.ConnOption.connect(ConnOption.java:133) ~[ojdbc6-11.2.0.3.jar!/:11.2.0.3.0]
at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:370) ~[ojdbc6-11.2.0.3.jar!/:11.2.0.3.0]
Tried to increase/tune CPU/memory allocation, connection pool (minidle, maxidle, maxactive). tried to disable the health checks, but then same failures were observed during runtime. The issue is observed randomly with no load on the system, mostly on first connection initialization, once connection is establishes it is stable. Since same DB is serving also old Openshift cluster without any issues, we have eliminated possible issue on DB side. The only difference between 2 environments is different Openshift installation and verison.
