"Fatal error occurred during protocol handshaking: Connection reset by peer" in Neo4j

Question

Running a Cypher batch statement (CALL apoc.periodic.iterate() in a @Query annotation of a Neo4jRepository), I encountered repeatedly the following error situation after a couple of hours of computation. Neo4j 3.4.9, Enterprise Edition.

Neo4j debug.log

2018-11-20 13:09:50.297+0000 ERROR [o.n.b.t.p.ProtocolHandshaker] Fatal error occurred during protocol handshaking: [id: 0x7035fd8f, L:/127.0.0.1:7687 - R:/127.0.0.1:56319] Connection reset by peer
java.io.IOException: Connection reset by peer
    at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
    at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
    at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
    at sun.nio.ch.IOUtil.read(IOUtil.java:192)
    at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
    at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288)
    at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1108)
    at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:345)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
    at java.lang.Thread.run(Thread.java:748)

Spring Boot context

org.neo4j.driver.v1.exceptions.ServiceUnavailableException: Connection to the database terminated. This can happen due to network instabilities, or due to restarts of the database
    at org.neo4j.driver.internal.util.ErrorUtil.newConnectionTerminatedError(ErrorUtil.java:45) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
    at org.neo4j.driver.internal.util.ErrorUtil.newConnectionTerminatedError(ErrorUtil.java:38) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
    at org.neo4j.driver.internal.async.inbound.ChannelErrorHandler.channelInactive(ChannelErrorHandler.java:76) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
    at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
    at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
    at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
    at org.neo4j.driver.internal.shaded.io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
    at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
    at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
    at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
    at org.neo4j.driver.internal.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:377) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
    at org.neo4j.driver.internal.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:342) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
    at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
    at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
    at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
    at org.neo4j.driver.internal.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:377) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
    at org.neo4j.driver.internal.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:342) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
    at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
    at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
    at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
    at org.neo4j.driver.internal.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:377) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
    at org.neo4j.driver.internal.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:342) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
    at org.neo4j.driver.internal.shaded.io.netty.handler.ssl.SslHandler.channelInactive(SslHandler.java:1010) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
    at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
    at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
    at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
    at org.neo4j.driver.internal.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1409) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
    at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
    at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
    at org.neo4j.driver.internal.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:927) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
    at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:822) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
    at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
    at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
    at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
    at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
    at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
    at java.lang.Thread.run(Thread.java:844) [?:?]

The application server as well as the Neo4j server are running on the same machine, so the error message assumption regarding "network instabilities" I asses as misleading.

What is the reason for the sudden loss of database connection to Neo4j, after everything works fine for hours? How can I fix the uncontrolled and hard termination of the Neo4j server?

If you had not already, could you activate the GC logs (https://neo4j.com/developer/kb/enabling-gc-logging/). I interpret "a couple of hours" as a large dataset. — meistermeier, Nov 22 '18 at 21:02
@meistermeier Thank you for the valuable hint! I will realize it. You are right with your assumption regarding the dataset: The relevant part consists of a few million rows for which a Cypher statement is executed. — ThirstForKnowledge, Nov 22 '18 at 21:19

score 1 · Accepted Answer · answered Nov 25 '18 at 12:26

If you also encounter the error scenario "Fatal error occurred during protocol handshaking: Connection reset by peer" in Neo4j, the following steps may help you to locate the origin:

activate and inspect the garbage collection logs (A big thank you goes out to @meistermeier)
check consistency of the database by bin/neo4j-admin check-consistency --database=graph.db more info

At the end of the day a missing transaction log was probably the reason for a corrupted database, which was triggered by a full hard disc. I had to delete and build up the database from the scratch. Since then everything is working fine again.

This error popped up randomly for me when trying to login on the neo4j browser on my small test database. Even after deleting all of the nodes and upgrading to 4.4.5 the error continues. The odd thing is that I can connect just fine using the neo4j Driver in my application. — Shy Albatross, Mar 26 '22 at 17:44

"Fatal error occurred during protocol handshaking: Connection reset by peer" in Neo4j

Neo4j debug.log

Spring Boot context

1 Answers1