0

I am using Docker 1.13 on Ubuntu 16.04 in Swarm Mode. I have one service (three replicas) published on port 80. I am seeing intermittent connection failures when trying to connect to the service.Initially the service works fine but once we run some load tests against the service these intermittent issues become very prevalent. The error rate can be as high as 50-75% of the requests fails, my theory is that the cluster becomes very unstable after the load tests.

The setup consists of 3 Manager Nodes and 3 worker nodes. I have a load balancer in front of the three worker nodes, for troubleshooting purpose I am using process of elimination and now I am on the worker node trying to hit the service using localhost (not going through the load balancer). At this point I have tried everything I can and about to give up, attributing the issue Docker Routing Mesh.

I am using overlay network with explicit subnet which does not overlap with other infrastructure items. VMs in Azure in a separate Virtual Network.

I have verified that the service instances are up and running using the command sudo docker service ps I have logged onto each individual Worker Node and verified the container/service instance is running and healthy using the command sudo docker ps

Please let me know if you need any other details or any suggestions.

Thanks!

isingh
  • 141
  • 1
  • 5

1 Answers1

0

I had the same problem with on premise setup, The problem exists in 1.12.x too. There are several github issues about that. You should cut the community some slack at the moment, they are doing an amazing job, swarm is pretty new and amazingly easy to setup.

What I did:

Moved to Kubernetes with flanneld for overlay networking, My test results are promising at the moment.

Farhad Farahi
  • 35,528
  • 7
  • 73
  • 70
  • Thanks for the response! Yes, no doubt community is doing an amazing job. I just wanted to get a confirmation if no solution so that we can figure plan our a plan on what can we do, either migrate to Kubernetes or I can do some tweaks. – isingh Feb 20 '17 at 15:06
  • No problem, I really didnt debug the issue after finding out about various related github issues, but if you did, make sure you create an issue with you r log files, so the community can fix it. – Farhad Farahi Feb 20 '17 at 15:13