0

System information:

  • Docker Engine Version: 17.06.2-ee-6

My team and I have been observing peculiar behaviour in our Docker cluster recently, where a REST request to a service (here: myservice) has failed randomly with a refused connection.

We removed and redeployed myservice for troubleshooting purposes. Suddenly, the number of requests to fail with a refused connection increased, allthough all containers were just recently started and healthy.

So, we ran host myservice from a container in the same network and got something like this:

myservice has address 172.10.13.241
myservice has address 172.10.0.81
Host myservice not found: 3(NXDOMAIN)

The result shows two IP addresses, and when testing the IP addresses directly (using curl for example) the address 172.10.13.241 works, while the address 172.10.0.81 always fails with a refused connection - explaining why we got a refused connection for 50% of the requests.

We then removed myservice again and ran host myservice once all containers stopped to find the following result:

myservice has address 172.10.0.81
Host myservice not found: 3(NXDOMAIN)

The broken IP address is still registered, allthough the service has been removed.

We further ensured that we weren't fooled by some DNS caching by restarting the container we ran the command from, but the result stayed the same.

Questions:

  • How can this happen?
  • Can I / How do I remove the orphaned DNS mapping from the Docker network?
  • What can we do to prevent this from happening?
Markus Appel
  • 173
  • 1
  • 6
  • Please include the command used to create the service (or compose file). Also note that your version is rather old and unpatched. Fixes for networking are missing from your install. – BMitch Jan 07 '20 at 14:43
  • Also, include a `docker network inspect -v ` on your respective network. – BMitch Jan 07 '20 at 14:45

0 Answers0