0

Randomly or after docker-compose down/up my application is not reachable on the container host ip.

I use NAT as docker network and from within the containers I can ping to other containers / internet / reach the ports that should be opened. If I do another docker compose down / up then it works fine again.

How do I troubleshoot this in a correct manner?

docker version 17.06.2-ee11 Windows server 2016 build 1709

Dresse
  • 473
  • 6
  • 15

1 Answers1

1

You may have run into a bug in the networking stack. I recently ran into the same issue, which showed the following additional oddities.

  • When the issue occurs, if you check the port with PowerShell Test-NetConnection, it will randomly switch between open and closed.
  • If you run a packet monitor on the request (e.g wireshark) you will see that the initial connection handshake often succeeds, but then something happens and you will start seeing multiple re-transmissions before the connection drops.

For us, the issue was actually appearing in a 5 node cluster. Suddenly random replicas of the container would stop responding on different nodes, with the node seeming to be different every time.

After spending almost a week ensuring we hadn't done anything to cause this, I logged a support request with Microsoft, and received confirmation yesterday (funny timing) that this was indeed a bug in the networking stack, which is currently still being investigated.

As such, if you have hit the same bug, at the moment it is still under investigation with no suggested work around, beyond what you already discovered of just restarting the container.

CodedBeard
  • 862
  • 8
  • 19
  • Thanks for your Reply CodedBeard. In the meanwhile I have traced the issue to the containers not being able to resolve hostnames. Ping works fine but on hostname it fails. Apparantly this was a known issue with workaround as explained in the following post: https://fluentbytes.com/how-to-work-around-docker-compose-dns-issue-with-docker-on-windows/ However that workaround doesn't work for me so I am back to square. I hope that MS will provide a fix for this issue soon, is there any way I can follow up as well? – Dresse May 24 '18 at 09:10