16

I have together 6 containers running in docker swarm. Kafka+Zookeeper, MongoDB, A, B, C and Interface. Interface is the main access point from public - only this container publish the port - 5683. The interface container connects to A, B and C during startup. I am using docker-compose file + docker stack deploy, each service has a name which is used as host for interface. Everything starts successfully and works fine. After some time (20 mins,1h,..) I am not able to make request to interface. Interface receives my requests but application lost connection with service A,B,C or all of them. If I restart interface, it's able to reconnect to services A,B,C.

I firstly thought it's problem of application so I expose 2 new ports on each service (interface, A,B,C) and connect with profiler and debugger to them. Application is running properly, no leaks, no blocked threads, normally working and waiting for connections. Debugger shows me that when I make a request to interface and interface tries to request service A, Connection reset by peer exception was thrown.

During this debugging I found out interesting stuff. I attached debugger to interface when the services started and also debugger was disconnected after some time. + I was not able to reconnect it, until I made request to the container -> application. PRoblem - handshake failed.

Another interesting stuff that I found out was that I was not able to request neither interface. So I used wireshark to see what's going on and: SYN - ACK was fine. Then application post some data and interface respond with FIN,ACK. I assume that this also happen when interface tries to request service A and it FIN the connection. Codebase of Interface, A,B and C is the same regarding netty server.

Finally, I don't think it's a application issue. Why? I tried to deploy containers not as services. I run each container separately, published the ports of each and endpoint of services were set to localhost. (not overlay network). And it is working. Containers run without problem. + I didn't say at the beginning, that the the java applications (interface, A,B,C) runs without problem when they are running as standalone application - not in docker.

Could you please help me what could be the issue? Why the docker in case of overlay network is closing sockets?

I am using newest docker. I used also older.

Ondrej Tomcik
  • 1,150
  • 9
  • 24
  • Is the "Interface", "A", "B", and "C" all your own Java applications? I am confused what "Interface" is. Also, have you tried this in Docker Compose without Docker Swarm (just single host)? My guess is that the application is working strangely with the Docker Swarm router so maybe trying without Docker Swarm might be a good idea. – Andy Shinn Apr 09 '17 at 23:41
  • Yes, A,B,C and interface are my java application, with same base code. Shared base-code contains among other things netty. Interface is only name for service to explain easily the issue - that only this interface is the public one and act as a gateway to other services, which do not publish ports. + It takes care of authentication and authorization, but that's the business logic. – Ondrej Tomcik Apr 10 '17 at 05:10
  • And as I wrote, I tried to run each container separately, each one published the ports it expose and I used localhost as ip so interface was connecting not to dns hostname but to localhost. This worked without any issue. – Ondrej Tomcik Apr 10 '17 at 05:16
  • I run the docker daemon in debug mode and when I did the request to interface and it was not working, I found in the log "Miss notification, l2 mac 02:42:0a:ff:00:02 DEBU[1479] Miss notification, l2 mac 02:42:0a:ff:00:08 DEBU[1479] Miss notification, l2 mac 02:42:0a:ff:00:08 DEBU[1479] Miss notification, l2 mac 02:42:0a:00:00:07 DEBU[1479] miss notification for dest IP, 10.0.0.2 . Any suggestions? – Ondrej Tomcik Apr 18 '17 at 13:00
  • Another interesting remark is that interface is connecting to Service A and in logs is: "Connected, Address: iotivity-accountserver/10.0.0.2:5685" . Okay, but when I inspect network, this container iotivity-accountserver has "IPv4Address": "10.0.0.3/24". And in network inspect, nowhere is mentioned 10.0.0.2 address. Also connection is lost to 10.0.0.2. – Ondrej Tomcik Apr 18 '17 at 13:25
  • Did you perform your checks on a swarm with only a single node? – gesellix Apr 19 '17 at 18:10
  • Yes, no change. – Ondrej Tomcik Apr 20 '17 at 19:52

2 Answers2

22

Finally, I was able to solve the problem.

What was happening, one more time. Interface opens permanent TCP connection to A,B,C. When you try to run these services A,B,C as a standalone java applications, everything is working. When we dockerize them and run in swarm, it was working only few minutes. Strange was that the connection between Interface and another service was interrupted in the moment when you made a request from client to interface.

After many many unsuccessful tests and debugging each container I tried to run each docker container separately, with mapped ports and as endpoint I specified localhost. (each container exposed ports and interface was connecting to localhost) Funny thing happen, it was working. When you run containers like this, different network driver for container is used. Bridge one. If you run it in swarm, overlay network driver is used.

So it had to be something with the docker network, not with application itself. Next step was tcpdump from each container after couple of minutes, when it should stop working. It was very interesting.

  • Client -> Interface (OK, request accepted)
  • Interface ->(forward request because it belongs to A) A
    • Interface -> A [POST]
    • A -> Interface [RESET]

A was reseting opened TCP communication after couple of minutes without communication. Why?

Docker uses IP Virtual Server and IPVS maintains its own connection table. The default timeout for CLOSE_WAIT connections in IPVS table is 60 seconds. Hence when the server sends something after 60 seconds, the IPVS connection is no longer available and the packet looks invalid for a new TCP session and gets RST. On the client side, the connection remains forever in FIN_WAIT2 state because the app still has the socket open; kernel's fin_wait timer kicks in only for orphaned TCP sockets.

This is what I read about it and how understand it. I am not sure if my explanation of problem is correct, but based on these assumptions I implemented ping-pong between Interface and A,B,C services in case there is no communication for <60seconds. And, it’s working.

Ondrej Tomcik
  • 1,150
  • 9
  • 24
  • Very useful research! Thank you. But I have a more deep problem here. One of my containers is MongoDB, and it randomly resets the connection and I obviously unable to implement a ping-pong here.. – Elessar.perm Jun 23 '17 at 09:09
  • I am using also MongoDB and I have no problem with it. I would try different client library. – Ondrej Tomcik Jun 25 '17 at 20:17
  • 1
    This is official client library by Mongo team and I managed to understand some conditions to reproduce. If my containers communicate through overlay network it works fine. If one is trying to communicate through published port (via ingress network) it has a risk. And this is not related to client library because I tried to up two official containers: one with the database and the second running `mongodump` (same image, just different startup command) and it works same way. It randomly resets connection when I use published port and it works fine whenever I use internal network address. – Elessar.perm Jun 25 '17 at 20:57
  • It don't break always though. I managed to mongodump a 20gb collection (12m of documents) through published port, but it almost always breaks when I'm dumping something like 200gb collection. – Elessar.perm Jun 25 '17 at 21:06
0

Got the same issue. Specified

endpoint_mode: dnsrr

to properties of the service which plays "server" role and it works just fine. https://forums.docker.com/t/tcp-timeout-that-occurs-only-in-docker-swarm-not-simple-docker-run/58179

Elijah Root
  • 11
  • 3