3

On AWS EC2, trying to connect to a private RDS instance from a Swarm container. It fails from within the container but works well from the host. I have a single master and node setup - 

bash-4.3# telnet mydb.cd1xokc1nbn4.ap-southeast-1.rds.amazonaws.com 5432
telnet: can't connect to remote host (10.0.2.3): Host is unreachable
bash-4.3# exit

ubuntu@ip-10-0-2-157:~/metabase$ telnet mydb.cd1xokc1nbn4.ap-southeast-1.rds.amazonaws.com 5432
Trying 10.0.2.3...
Connected to mydb.cd1xokc1nbn4.ap-southeast-1.rds.amazonaws.com.
Escape character is '^]'.

I am quite sure it's something related to networking in Swarm since if I do a docker run, I am able to connect to the DB.

Docker version - 

Client:
 Version:           18.09.0
 API version:       1.39
 Go version:        go1.10.4
 Git commit:        4d60db4
 Built:             Wed Nov  7 00:49:01 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.0
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.4
  Git commit:       4d60db4
  Built:            Wed Nov  7 00:16:44 2018
  OS/Arch:          linux/amd64
  Experimental:     false
vivekyad4v
  • 13,321
  • 4
  • 55
  • 63

2 Answers2

1

I was able to make it work by using the host network mode. 

    .......
    networks:
      - metabase

networks:
  metabase:
    external:
      name: "host"

PS - It was working fine after using host network mode but that's not suggested. More answers are welcome since I am not sure about the exact reason as thought why it was working with default swarm virtual network.

vivekyad4v
  • 13,321
  • 4
  • 55
  • 63
1

I had the same problem - based on the IP address you wanted to connect to I guess that your problem had the same cause.

Docker Swarm overlay networks by default use IP addresses in a 10.0.x.x range. In your case the service/container which you wanted to connect to the database was likely in a Docker network in the 10.0.2.x IP range - so it would try to connect to the database host via that network, which obviously won't work.

So you need to resolve this conflict. One way is to configure the Swarm to use a different IP range. The downside is that you can only do this when initializing the Swarm (so you would have to recreate it). For example:

docker swarm init --default-addr-pool 10.10.0.0/16

One way to verify that it works as expected is creating a new network afterwards and checking the subnet:

docker network create -d overlay proxy
docker network inspect proxy | grep Subnet

If you have control over the IP ranges for your AWS subnets, you could probably also change the IP address ranges there, to avoid the problem that way.

stempler
  • 750
  • 6
  • 15
  • 1
    Thank you, almost spent an entire day without a solution. I was able to leave the docker swarm and create a new one. This resolved the issue because my AWS subnets were also created under the following CIDR block range: 10.0.0.0/16 – Nirojan Selvanathan Feb 03 '22 at 09:05