0

We are trying to integrate the HikariCP (release 2.4.4) into our application. After some time of usage the pool fails to acquire new connections throwing:

    java.lang.NullPointerException
        at com.zaxxer.hikari.pool.PoolBase.setNetworkTimeout(PoolBase.java:464) ~[HikariCP-2.4.4.jar:?]
        at com.zaxxer.hikari.pool.PoolBase.isConnectionAlive(PoolBase.java:131) ~[HikariCP-2.4.4.jar:?]
        at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:171) ~[HikariCP-2.4.4.jar:?]
        at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:147) ~[HikariCP-2.4.4.jar:?]
        at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:83) ~[HikariCP-2.4.4.jar:?]

The jdbc driver we are using is ojdbc-7 with version 12.1.0.2. The pool uses following configuration:

    allowPoolSuspension.............false
    autoCommit......................false
    catalog.........................null
    connectionInitSql..............."BEGIN EXECUTE IMMEDIATE 'SET ROLE SOME_ROLE IDENTIFIED BY SOME_PASSWORD '; END;"
    connectionTestQuery............."SELECT 1 FROM DUAL"
    connectionTimeout...............15000
    dataSource......................null
    dataSourceClassName.............null
    dataSourceJNDI..................null
    dataSourceProperties............{v$session.machine=host, password=<masked>, v$session.program=my application}
    driverClassName................."oracle.jdbc.OracleDriver"
    healthCheckProperties...........{}
    healthCheckRegistry.............null
    idleTimeout.....................60000
    initializationFailFast..........true
    isolateInternalQueries..........false
    jdbc4ConnectionTest.............false
    jdbcUrl........................."jdbc:oracle:thin:@//host.company.com:1521/database.company.com"
    leakDetectionThreshold..........1800000
    maxLifetime.....................0
    maximumPoolSize.................18
    metricRegistry..................null
    metricsTrackerFactory...........null
    minimumIdle.....................2
    password........................<masked>
    poolName........................"TEST_POOL"
    readOnly........................false
    registerMbeans..................false
    scheduledExecutorService........null
    threadFactory...................null
    transactionIsolation............null
    username........................"USER_NAME"
    validationTimeout...............5000

Is it a bug or a missconfiguration?

sebastian
  • 2,427
  • 4
  • 32
  • 37
  • Are you `close`ing the connections to return them to the pool when you're done with them? – OldCurmudgeon Mar 07 '16 at 09:57
  • @OldCurmudgeon Yes, mostly we are calling simply connection.close(). Sometimes we are additionally using evictConnection before calling close. – sebastian Mar 07 '16 at 10:04
  • Mostly??? You should `close` in a `finally` so it **always** gets released to the pool. Probably not a good idea to use `evictConnection` without good reason - why do you do that? – OldCurmudgeon Mar 07 '16 at 10:07
  • @OldCurmudgeon Calling close is guaranteed (mostly without calling evictConnection). We have a system specific tenant system. When a tenant has been initialized on a session once it is not possible to reuse that session for another tenant. For that reason we need to evict that connection. – sebastian Mar 07 '16 at 10:11

1 Answers1

0

I'm not 100% sure, but it looks like either:

  • You are evicting a connection after you have called close(), which is not allowed. Or..
  • You are evicting a connection and then calling close(), which is not allowed.

When you evict a connection, you must be the owner of that connection (obtained from getConnection(), and subsequently you must not close() the connection (it will be closed automatically). And as explained above, if you have called close() already, the connection is already back in the pool and it is not valid to evict it, as you are no longer the owner.

EDIT: Let me be clearer. From studying how this exception could be reached it seems clear that you are first closing the connection, and secondly evicting the connection. The reverse (evict and then close) would not result in this error.

This is known as "use after return" and is similar to "use after free" bugs in languages without garbage collection. When you close a connection, it is returned to the pool. From that instant the connection is available to be claimed by another thread -- the caller of close() is no longer the owner.

This is exactly analogous to calling free() on memory in C/C++. Instantly after doing so the memory is available to be claimed by another caller -- the caller of free() is no longer the owner. In the C/C++ case, if you continue to use a reference to the freed memory you risk corrupting data of another thread that has now allocated it.

In the case of nearly any pooling library in Java (connection or otherwise), once you release an object back to the pool you are no longer the owner. Nothing can prevent you from retaining a reference to the returned object.

In this case, once you have called close(), the object is returned to the pool instantly. If another thread obtains the connection legally from the pool (getConnection()), while at the same time the previous owner calls evict(), you will easily run into this issue.

We may choose to harden this code path (or not). HikariCP is not particularly paternalistic philosophically; favoring documentation over code. For example, if you pass a null into evict() you will be met with an NPE somewhere. Could we check for null and ignore it? Sure. Multiply that approach across the codebase and it could easily grow by 20%. Or, how about don't do that, developer?

It is a fairly simple contract:

  • You can only evict a connection that you own.
  • As soon as you have closed a connection, you no longer own it.
brettw
  • 10,664
  • 2
  • 42
  • 59
  • Your second point is correct, we are always calling close on a connection even in that case it has been evicted. Can you state the resource of this information? I can't find anything about it in documentation or in the javadocs. And this bevaviour seems not very obvious to me. – sebastian Mar 07 '16 at 13:45
  • My understanding is, that evictConnection is marking the connection for eviction but not closing it directly. – sebastian Mar 07 '16 at 13:54
  • @sebastan your reading of evictConnection is also correct. NPE is a bug to find and be fixed in hikaricp. – Nitin Mar 07 '16 at 14:51
  • HikariDataSource.evictConnection() will immediately evict a connection, not merely mark it. HikariPool.evictConnection() will be called, which will call softEvictConnection() with ``owner`` set to *true*, which will call closeConnection(), rather than marking it. I do not consider the NPE a bug, but I agree that the documentation of ``evictConnection()`` needs to be made crystal clear about its usage. – brettw Mar 08 '16 at 05:17
  • Beside the fact the documentation is not obvious in that point. Receiving a NPE when such an illegal state is reached is not very helpful either. Shouldn't there be an `IllegalStateException` immediatly after calling `close` when `evictConnection` has already been called? I also want to add for consideration that the call of `evictConnection` may be located at a higher level of the application structure as the calling of `close`. This forces the programmer to reach this information down which is not very comfortable. – sebastian Mar 08 '16 at 06:03
  • Stacktrace suggests that this is a bug as it IS pool, NOT 'user' who is passing null connection for validation. – Nitin Mar 08 '16 at 08:51
  • @sabastian Just so you know, we committed a "fix" for this issue. You can now call ``close()`` and ``evict()`` in any order without error. Note, if your intention expressed above is using eviction to prevent a connection being shared with another tenant, it will not work as you expect. As soon as you call ``close()`` that connection can be acquired. Calling ``evict()`` at that point will merely mark the connection for eviction, but clearly the new thread acquirer can continue to use the connection. Only by calling ``evict()`` while owning the connection can you prevent reuse. – brettw Mar 08 '16 at 15:08