1

We are running a CXF 2.7.11 application on WAS 8.5.5.2 server. Application has classloading parent last property also we disabled IBM JaxWS engine as instructed on CXF documentation.

Application is running fine a couple of days, after that we get below exceptions and TCP channel seems to be full.

From the stack trace that have ws classes I suspect CXF for this problem but that may be a result of another problem

The application is also a Spring MVC application that exposes REST resources..

[10.11.2014 05:00:20:887 EET] 00000049 TCPChannel    W   TCPC0004W: TCP Channel TCP_2 has exceeded the maximum number of open connections 20000.
[10.11.2014 05:02:16:343 EET] 0000023f SSLHandshakeE E   SSLC0008E: Unable to initialize SSL connection.  Unauthorized access was denied or security settings have expired.  Exception is javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
    at com.ibm.jsse2.b.a(b.java:56)
    at com.ibm.jsse2.nc.a(nc.java:90)
    at com.ibm.jsse2.nc.unwrap(nc.java:292)
    at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:26)
    at com.ibm.ws.ssl.channel.impl.SSLConnectionLink.readyInbound(SSLConnectionLink.java:535)
    at com.ibm.ws.ssl.channel.impl.SSLConnectionLink.ready(SSLConnectionLink.java:295)
    at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214)
    at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113)
    at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
    at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
    at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
    at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
    at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
    at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
    at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
    at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1864)
kolossus
  • 20,559
  • 3
  • 52
  • 104
iesen
  • 65
  • 2
  • 11
  • well it seemed that heap for the JVM was very low and this may be the side effect of the low memory. We did not see error since increasing max heap size. – iesen Dec 10 '14 at 08:45

2 Answers2

0

So this is a bit tricky. You can simply increase the number of connections which can be done from the console:

Servers > WebSphere Application Servers > SERVER_NAME > web container > web container transport chains > TCP CHANNEL

The reason I said this is tricky because there could be a larger underlying issue, for example, a connection leak. To get to the point where you are using up 20K connections is quite a lot, however, I don't know how much load you're expecting on this server. If this is simply a test environment then you need to start looking into a possible connection leak. 

Unable to initialize SSL connection. Unauthorized access was denied or security settings have expired. Exception is javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?

This portion of the error message means that plain-text non-SSL connections are being made to an SSL port. You might also want to take a look at that and see who's making these calls because it's an overhead.

wFateem
  • 346
  • 1
  • 2
  • 11
0

Using 20000 connections is extremely high. You probably have bugs in your client code which is leaking connections. If you are using CXF in client you may take a look at this https://issues.apache.org/jira/browse/CXF-5144.

Increasing connections number will not solve your issue, it will just delay it.

Gas
  • 17,601
  • 4
  • 46
  • 93
  • They say it is fixed on 2.7.7, but i will also try to upgrade to 3.0.2 – iesen Nov 11 '14 at 20:02
  • Well you say that the bug is not in my CXF service application, but some client code is buggy. How can i determine the leaky client? The application is also a Spring MVC application that exposes REST resources – iesen Nov 11 '14 at 20:27
  • @iesen You could check the netstat -an and see if you will see some established connections growing. Maybe you will be able to see client IP and it will give you some clues. – Gas Nov 11 '14 at 20:30