1

I'm getting tons of these errors deploying a grpc (1.5.0) java app on marathon.

Aug 01, 2017 4:49:12 PM io.grpc.netty.NettyServerTransport notifyTerminated
SEVERE: Transport failed
java.io.IOException: Connection reset by peer
    at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
    at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
    at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
    at sun.nio.ch.IOUtil.read(IOUtil.java:192)
    at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
    at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288)
    at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1100)
    at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:372)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
    at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
    at java.lang.Thread.run(Thread.java:745)

EDIT:

I'm also noticing

[pool-1-thread-4] --- 2017-08-02 12:31:09,465 WARN  e.a.s.grcp.ConnectionStatusServer - requests.size 1
[nioEventLoopGroup-2-1] --- 2017-08-02 12:31:09,465 DEBUG i.n.h.c.http2.Http2ConnectionHandler - [id: 0x0a5278e3, L:0.0.0.0/0.0.0.0:31169 ! R:/10.124.56.175:47384] Sending GOAWAY failed: lastStreamId '7', errorCode '2', debugData 'Connection reset by peer'. Forcing shutdown of the connection.java.io.IOException: Broken pipe
 --->   at sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
 --->   at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51)
 --->   at sun.nio.ch.IOUtil.write(IOUtil.java:148)
 --->   at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:504)
 --->   at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:432)
 --->   at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:856)
 --->   at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:362)
 --->   at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:823)
 --->   at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1296)
 --->   at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776)
 --->   at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768)
 --->   at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749)
 --->   at io.netty.handler.codec.http2.Http2ConnectionHandler.onError(Http2ConnectionHandler.java:573)
 --->   at io.grpc.netty.AbstractNettyHandler.exceptionCaught(AbstractNettyHandler.java:84)
 --->   at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:285)
 --->   at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:264)
 --->   at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:256)
 --->   at io.netty.channel.DefaultChannelPipeline$HeadContext.exceptionCaught(DefaultChannelPipeline.java:1301)
 --->   at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:285)
 --->   at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:264)
 --->   at io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:914)
 --->   at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:104)
 --->   at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:145)
 --->   at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
 --->   at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
 --->   at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
 --->   at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
 --->   at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
 --->   at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
 --->   at java.lang.Thread.run(Thread.java:745)
 --->
mfirry
  • 3,634
  • 1
  • 26
  • 36
  • Great question - please update this post as you gain more insight. While evaluating grpc-java I ran into the same issue, unrelated to Marathon. The problem popped up more when using a direct executor, but was not exclusive to it. I assume it's relatively benign (just indicating dropped connections), but not sure. In my opinion this issue needs to be addressed by the grpc-java framework as a regular use case, without leaving a fat stacktrace in the logs. I imagine that you'd get good feedback from the grpc Google Group if nothing pops up here. – Uli Aug 01 '17 at 20:12
  • The bottom line why you're seeing the exception is that something is tearing down your (persistent) TCP connection. Could be something in between (e.g. connection getting closed by firewall after timeout), or even a misbehaving client (e.g. client crash). grpc-java should just handle it in a non-alarming manner, because this is a very common occurrence. – Uli Aug 01 '17 at 20:30
  • Thanks a lot @Uli. I'm trying to bypass marathon-lb and see if this is till happening. – mfirry Aug 02 '17 at 06:37
  • The problem of seeing SEVERE log messages for reset connections is documented and the severity of the log messages apparently will be dropped in the next release: https://github.com/grpc/grpc-java/issues/1768 The issue also implies that the effect of broken connections is generally relatively benign. (would be handled by grpc-java via channel reconnects, etc., if you set it up correctly) – Uli Aug 05 '17 at 21:02

0 Answers0