Connection reset by peer with async http client and netty

Question

I've got a strange problem with Async http client with netty as the http provider, it's a play application calling a remote web service which we don't have access to. It's also important to note that the service is on a SSL-encrypted domain and only reponds to this domain and not the IP. Stacktrace:

java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.7.0_60]
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[na:1.7.0_60]
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.7.0_60]
at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[na:1.7.0_60]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) ~[na:1.7.0_60]
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64) ~[netty-3.9.2.Final.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) ~[netty-3.9.2.Final.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) ~[netty-3.9.2.Final.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) ~[netty-3.9.2.Final.jar:na]
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[netty-3.9.2.Final.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_60]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_60]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_60]

The most strange is that this happened suddenly (it's a test environment used by a customer) which makes me believe something is up with either the receiving service or the network, but I can't find out whats wrong because calling the same url with curl works fine.

I've tried to isolate the problem by using the async http client (which play uses for it's WS interface) locally and get the same error, curl works, using javas URL-api also works. Also when I'm using JDKAsyncHttpProvider as the http provider it also works perfectly fine.

   try {
        AsyncHttpClientConfig cf = new AsyncHttpClientConfig.Builder().build();
        // works if I use JDKAsyncHttpProvider..
        //AsyncHttpClient asyncHttpClient = new AsyncHttpClient(new JDKAsyncHttpProvider(cf));
        AsyncHttpClient asyncHttpClient = new AsyncHttpClient(cf);

        Future<Response> f = asyncHttpClient.prepareGet(nw).execute();
        Response r = f.get();

        System.out.println(r.getResponseBody());
    } catch(Exception e) {
        e.printStackTrace();
    }

Since I'm using play framework I can't change the provider.

I'm currently stuck and I'm asking for pointers what the problem might be or the next step in my debugging.

Update: with debugging options all on I get the following error:

Extension ec_point_formats, formats: [uncompressed]
***
[write] MD5 and SHA1 hashes:  len = 149
0000: 01 00 00 91 03 01 53 AD   17 DD 91 78 24 7E 9A 58  ......S....x$..X
0010: 0E CA D9 ED 22 5A A9 DF   3F 4C DD 25 58 E1 E6 C1  ...."Z..?L.%X...
0020: 4C 63 3E 11 78 1E 00 00   2A C0 09 C0 13 00 2F C0  Lc>.x...*...../.
0030: 04 C0 0E 00 33 00 32 C0   07 C0 11 00 05 C0 02 C0  ....3.2.........
0040: 0C C0 08 C0 12 00 0A C0   03 C0 0D 00 16 00 13 00  ................
0050: 04 00 FF 01 00 00 3E 00   0A 00 34 00 32 00 17 00  ......>...4.2...
0060: 01 00 03 00 13 00 15 00   06 00 07 00 09 00 0A 00  ................
0070: 18 00 0B 00 0C 00 19 00   0D 00 0E 00 0F 00 10 00  ................
0080: 11 00 02 00 12 00 04 00   05 00 14 00 08 00 16 00  ................
0090: 0B 00 02 01 00                                     .....
New I/O worker #2, WRITE: TLSv1 Handshake, length = 149
[Raw write]: length = 154
0000: 16 03 01 00 95 01 00 00   91 03 01 53 AD 17 DD 91  ...........S....
0010: 78 24 7E 9A 58 0E CA D9   ED 22 5A A9 DF 3F 4C DD  x$..X...."Z..?L.
0020: 25 58 E1 E6 C1 4C 63 3E   11 78 1E 00 00 2A C0 09  %X...Lc>.x...*..
0030: C0 13 00 2F C0 04 C0 0E   00 33 00 32 C0 07 C0 11  .../.....3.2....
0040: 00 05 C0 02 C0 0C C0 08   C0 12 00 0A C0 03 C0 0D  ................
0050: 00 16 00 13 00 04 00 FF   01 00 00 3E 00 0A 00 34  ...........>...4
0060: 00 32 00 17 00 01 00 03   00 13 00 15 00 06 00 07  .2..............
0070: 00 09 00 0A 00 18 00 0B   00 0C 00 19 00 0D 00 0E  ................
0080: 00 0F 00 10 00 11 00 02   00 12 00 04 00 05 00 14  ................
0090: 00 08 00 16 00 0B 00 02   01 00                    ..........
09:06:05.883 [New I/O worker #2] DEBUG c.n.h.c.p.n.NettyAsyncHttpProvider - Unexpected I/O exception on channel [id: 0x9820cfc1, /192.168.101.123:41411 => domain/148.136.157.62:443]
java.io.IOException: Connection reset by peer

So something is up with the TLS handshake, is there something more I can do when debugging this issue without access to the remote service?

Update again, got a new error message this time:

[write] MD5 and SHA1 hashes:  len = 149
0000: 01 00 00 91 03 01 53 AD   1C 54 D6 57 05 7A 89 E5  ......S..T.W.z..
0010: 55 60 01 2F 48 A5 BA EB   4C E6 96 68 E7 B8 2B 67  U`./H...L..h..+g
0020: A2 4C AF C8 9D 10 00 00   2A C0 09 C0 13 00 2F C0  .L......*...../.
0030: 04 C0 0E 00 33 00 32 C0   07 C0 11 00 05 C0 02 C0  ....3.2.........
0040: 0C C0 08 C0 12 00 0A C0   03 C0 0D 00 16 00 13 00  ................
0050: 04 00 FF 01 00 00 3E 00   0A 00 34 00 32 00 17 00  ......>...4.2...
0060: 01 00 03 00 13 00 15 00   06 00 07 00 09 00 0A 00  ................
0070: 18 00 0B 00 0C 00 19 00   0D 00 0E 00 0F 00 10 00  ................
0080: 11 00 02 00 12 00 04 00   05 00 14 00 08 00 16 00  ................
0090: 0B 00 02 01 00                                     .....
New I/O worker #2, WRITE: TLSv1 Handshake, length = 149
[Raw write]: length = 154
0000: 16 03 01 00 95 01 00 00   91 03 01 53 AD 1C 54 D6  ...........S..T.
0010: 57 05 7A 89 E5 55 60 01   2F 48 A5 BA EB 4C E6 96  W.z..U`./H...L..
0020: 68 E7 B8 2B 67 A2 4C AF   C8 9D 10 00 00 2A C0 09  h..+g.L......*..
0030: C0 13 00 2F C0 04 C0 0E   00 33 00 32 C0 07 C0 11  .../.....3.2....
0040: 00 05 C0 02 C0 0C C0 08   C0 12 00 0A C0 03 C0 0D  ................
0050: 00 16 00 13 00 04 00 FF   01 00 00 3E 00 0A 00 34  ...........>...4
0060: 00 32 00 17 00 01 00 03   00 13 00 15 00 06 00 07  .2..............
0070: 00 09 00 0A 00 18 00 0B   00 0C 00 19 00 0D 00 0E  ................
0080: 00 0F 00 10 00 11 00 02   00 12 00 04 00 05 00 14  ................
0090: 00 08 00 16 00 0B 00 02   01 00                    ..........
Hashed wheel timer #2, called closeOutbound()
Hashed wheel timer #2, closeOutboundInternal()
Hashed wheel timer #2, SEND TLSv1 ALERT:  warning, description = close_notify
Hashed wheel timer #2, WRITE: TLSv1 Alert, length = 2
Hashed wheel timer #2, called closeInbound()
Hashed wheel timer #2, fatal error: 80: Inbound closed before receiving peer's close_notify: possible truncation attack?
javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?
Hashed wheel timer #2, SEND TLSv1 ALERT:  fatal, description = internal_error
Hashed wheel timer #2, Exception sending alert: java.io.IOException: writer side was already closed.

Updated, resolved:

The remote host configured the https-certificate to use Server Name Indication, SNI which caused the netty provider to fail. I guess the netty provider doesn't support SNI yet.

"Connection reset by peer" means the remote end of the connection closed it. You need to find out why the remote software is dropping the connection. — Kenster, Jun 26 '14 at 11:24
That I've found out, but why do I get the error only when I'm using the Netty provider and not when I'm using curl in the terminal or as in my example with JDKAsyncHttpProvider? — r0stig, Jun 26 '14 at 11:38
I don't know, but finding out why the remote software is dropping the connection would be a big clue. — Kenster, Jun 26 '14 at 11:40
The remote service is owned by another company and the only thing I got out of them is that nothing is logged (I think they're using IIS with .Net webAPI). I guess what I'm asking is if I can make sure it's not a fault on my end. — r0stig, Jun 26 '14 at 11:45

score 2 · Answer 1 · answered Jun 27 '14 at 00:34

When you say "SSL encrypted domain", you mean HTTPS?

If so, you can use the debugging feature in WS SSL: http://www.playframework.com/documentation/2.3.x/DebuggingSSL

or set JSSE's debug flags directly with -Djavax.net.debug=all.

Also, turn on logger.play.api.libs.ws=DEBUG and logger.org.asynchttpclient=DEBUG in application.conf for maximum verbosity.

You can also change the provider directly, if you're willing to use AsyncHttpClient directly, without going through WS. This will mean you may have to mess around with a Promise in the onSuccess() callback, but you create a limited WS-like interface that will return you back a Future if this is a problem.

Thanks, I got some more interesting debug output. I updated the question. — r0stig, Jun 27 '14 at 07:23

score 0 · Answer 2 · answered Oct 03 '14 at 04:29

0

I had the same problem. I was able to resolve it by switching from Java 6 to Java 7. It is a problem with Server Name Indication (SNI) not being supported in Java 6, but is supported in Java 7.

answered Oct 03 '14 at 04:29

Roger Chang

3
1

Connection reset by peer with async http client and netty

2 Answers2