Timeout with JMX monitoring on cipher suites, connecting with Ssl

Question

Okay, so I have been working on this for a while trying to figure this out. I have JMX monitoring working from one client to the server, but not from another client to the server. To me, the only difference seems to be that one is RHEL6, the other RHEL7. It works with RHEL6.

So here is the relevant code, in obtaining the MBeanServerConnection:

env.put(JMXConnector.CREDENTIALS, credentials);
env.put(RMIConnectorServer.RMI_SERVER_SOCKET_FACTORY_ATTRIBUTE ,
    new SslRMIServerSocketFactory());
url = new JMXServiceURL(
    "service:jmx:rmi:///jndi/rmi://123.123.123.123:1234/jmxrmi");
jmxc = JMXConnectorFactory.connect(url, env);
mbsc = jmxc.getMBeanServerConnection();

I compile with an identical jdk1.8.0_51 versions locally in /opt on both machines. The keystores and truststores I created were first created on RHEL7, where this only works from RHEL6. I set these properties at runtime (formatted for readability):

/opt/java/jdk1.8.0_51/bin/java -cp . 
-Djavax.net.ssl.keyStore=/path/to/keyStore.key 
-Djavax.net.ssl.keyStorePassword=keypassword 
-Djavax.net.ssl.trustStore=/path/to/trustStore.key 
-Djavax.net.ssl.trustStorePassword=trustpassword 
-Djavax.net.debug=all  
MonitoringProgram

So what happens is that from my RHEL7 machine and the debugging on, I get plenty of output and then it freezes after this last output:

Ignoring unavailable cipher suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA256
Ignoring unavailable cipher suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA
Ignoring unavailable cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
Ignoring unavailable cipher suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

When I have to kill the program, or wait for an immensely long timeout. Now this same program, with debugging, has all of the exact same output (I've compared them with diff) until then, but continues instead of freezing:

Ignoring unavailable cipher suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA256
Ignoring unavailable cipher suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA
Ignoring unavailable cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
Ignoring unavailable cipher suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
Allow unsafe renegotiation: false
Allow legacy hello messages: true
...

With lots of more output due to the debug=all, and with my expected output (for now, simply trying to obtain the domains of the mbean server).

So what could be causing this difference? The worst part is the connection just freezes, rather than throwing me an exception that I might be able to debug. I have tried recreating the keystore a couple of times, but I don't think it's that anymore. I thought it might be due to the unavailable cipher suites, but the working instance has those same unavailable cipher suites before settling on this cipher suite: TLS_DHE_DSS_WITH_AES_128_CBC_SHA256 with TLSv1.2. Remember, it has the exact same output until it freezes, so I'm not sure if the debug tool is going to let me completely solve this.

Any ideas or input would be greatly appreciated.

Edit: I'm adding the connection timeout stacktrace, should have added that in the first place. Still working on a solution.

java.rmi.ConnectException: Connection refused to host: 123.123.123.123; nested exception is:
        java.net.ConnectException: Connection timed out
        at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
        at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
        at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
        at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:130)
        at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
        at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2432)
        at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
        at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
        at MonitoringProgram.getMBeanServerConnection(MonitoringProgram.java:173)
        at MonitoringProgram.main(MonitoringProgram.java:208)
Caused by: java.net.ConnectException: Connection timed out
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
        at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:427)
        at sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:88)
        at javax.rmi.ssl.SslRMIClientSocketFactory.createSocket(SslRMIClientSocketFactory.java:121)
        at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
        ... 9 more
Error: Connection refused

If things are really that serious, you could do a wireshark analysis on the communication? Even without understanding the whole conversation between client and server you might be able to spot at least some extra info? Like who waits for the other and such. — Gergely Bacso, Aug 17 '15 at 19:31
The fact that your servers are on different OS suggest that there could be several other differences in there setup as well. Have you verified that both of your instances are: - using the same java version - have the same networking parameters - have file permissions properly configured (keystore!) — Gergely Bacso, Aug 17 '15 at 19:32
Not very likely, but is it possible that you have two different security providers on the two clients? Is the first element of the Provider[] array returned by Security.getProviders() the same? — Dimitar, Aug 17 '15 at 19:33
@Gergely Permissions and even user accounts are identical. I'll try checking out the wireshark analysis. — briank, Aug 17 '15 at 19:49
When you say it works "with debugging", do you mean it works on RHEL7 with debugging, but freezes without debugging. or do you mean it works on RHEL6 with debugging, but freezes on RHEL7 without debugging? If the latter, perhaps setting debugging on changes the timing so the issue isn't surfaced, suggesting that it may be a threading issue in either your code or the JVM. — Warren Dew, Aug 17 '15 at 20:21
Is "immensely long" one minute? The next thing after "Ignoring cipher ..." is the TCP connection, and if the TCP connection gets no response it times out, IME on CentOS~RedHat after almost exactly one minute. Check `netstat -tn` for `SYN_SENT` (but not `ESTABLISHED`) to the target address&port. If this is the problem, check any firewall on the target such as `iptables` and anything between the source and target like a router or firewall. PS: "Ignoring cipher ... AES256 ..." is because you don't have "Unlimited Strength Policy" installed, but you don't need it, AES128 is plenty good. — dave_thompson_085, Aug 17 '15 at 22:22

Timeout with JMX monitoring on cipher suites, connecting with Ssl

0 Answers0